A file system synchronization method and system based on a vectorizable summarization algorithm

Abstract

The application relates to a file system synchronization method and system based on a vectorizable abstract algorithm, and belongs to the technical field of file system processing. The steps comprise the following: (1) for a file directory to be synchronized, an abstract tree is constructed for the component structure of the file directory by using an abstract algorithm; (2) the file directory with the generated abstract tree and the files thereunder are stored on a remote server, and a synchronization instruction is sent to a terminal; (3) the terminal performs file directory synchronization; and (4) daily maintenance of the synchronized directory. The application defines a set of vectorizable abstract algorithms, and uses the abstract algorithms to construct abstract trees, which are applied in the synchronization process of a file system. By converting data into vectors for calculation, the parallel processing capacity of a modern computer can be fully utilized, and the speed of abstract generation is significantly improved.

Description

TECHNICAL FIELD

[0001] The application relates to a file system synchronization method and system based on a vectorizable digest algorithm and belongs to the technical field of file system processing. BACKGROUND

[0002] A digest algorithm, also known as a hash algorithm or a hash algorithm, is a kind of algorithm that converts input data of an arbitrary length into a fixed length return value through a series of calculation methods and rules, and the value is called a hash value (hash value). The main features of the digest algorithm include compression, easy calculation, modification resistance, strong collision resistance and irreversibility, and are commonly used for data integrity verification, digital signature and the like.

[0003] Vectorization is a method for accelerating calculation through parallel processing technology. Modern processors have widely supported vectorization acceleration instruction sets, such as SSE, AVX of the x86 architecture, NEON, SVE of the ARM architecture and Vector extension instructions of the RISC-V architecture. Through the vectorization instruction set, multiple data can be operated simultaneously, thereby improving the calculation efficiency and accelerating the algorithm operation.

[0004] In computer science, the directory structure is an important part of the file system, which determines the organization of files on the disk. The tree directory structure is a common directory structure that simulates the tree structure in real life and has the advantages of clear hierarchy, easy management, high access efficiency and the like. The tree directory structure, also known as hierarchical structure or tree structure, is a directory structure in which each directory (including subdirectories) can contain files and other subdirectories, starting from the root directory. The biggest advantage of the tree directory structure is good scalability and flexibility, which can facilitate the addition, deletion, movement or renaming of files and directories.

[0005] The digest tree is a tree structure formed by combining the directory structure of the file system and the digest values of the files in the directory. The organization of the tree structure is consistent with the directory structure of the file system, except that the files in the directory are represented by the digest values of the files in the digest tree. For a directory, a digest file is created to record the digest values of all files under the directory. The digest value of the digest file is used as the digest value of the directory and is counted by the upper directory. However, there is a lack of effective file system synchronization method in the prior art, and the digest generation speed is slow. Therefore, the application is proposed. SUMMARY

[0006] In view of the deficiencies of the prior art, the application provides a file system synchronization method and system based on a vectorizable digest algorithm.

[0007] The technical scheme of the application is as follows:

[0008] A file system synchronization method based on vectorable abstract algorithm, steps as follows:

[0009] (1) For the file directory to be synchronized, abstract tree is constructed for the component structure of file directory using abstract algorithm;

[0010] (2) The file directory with generated abstract tree and the subordinate files are stored on remote server, and synchronization instruction is sent to terminal;

[0011] (3) Terminal carries out file directory synchronization;

[0012] (4) Daily maintenance of synchronized directory.

[0013] Through the above process, the directory in file system can be safely and reliably synchronized with small cost. The consistent abstract value of local file and server file can guarantee the consistency of files, and the consistent abstract tree of local directory and server directory can guarantee the consistency of two directories, thus the directory to be synchronized in file system can also be reliably guaranteed.

[0014] According to the application, preferably, in step (1), the abstract tree construction process comprises file directory processing and file processing in directory, and specifically as follows:

[0015] File directory: the file directory comprises an abstract file for describing the abstract value of file in the file directory and subordinate directory, wherein the abstract value of abstract file is taken as the abstract value of file directory;

[0016] File: the abstract value of file is calculated using abstract algorithm, and the name of file and the calculated abstract value are stored in the abstract file in the directory of file;

[0017] Abstract tree: the tree structure of abstract value is formed by the abstract value of each subdirectory / file in folder and the tree structure of directory in file system.

[0018] According to the application, preferably, in step (1), the abstract algorithm step is as follows:

[0019] ① Firstly, 8 linear independent secret key vectors K1, K2……K8 are selected, and the original data is cut into independent data blocks D1, D2……Dn with 8 bytes as a unit (the part less than 8 bytes is filled with 0xFF, 0xFF is a number in hexadecimal system, 1 byte is expressed by 8 bit, and 8 bits all are 1 in binary system, which is 11111111, and converted into 16 hexadecimal system, which is 0xFF);

[0020] The independent data blocks of D1, D2,..., Dn are regarded as vectors and multiplied with the key vector K1 to obtain F1K1, F2K1,..., FnK1, and the obtained vector products are collected to obtain MK1 (accumulation, the high data of each dimension is discarded, and only one byte is reserved), and similarly, the vector products with the key vectors K2, K3,..., K8 are obtained, and the results are collected to obtain MK2, MK3,..., MK8;

[0021] The high data is the data overflowed when the product is calculated, for example, 200*200=40000, the unsigned hexadecimal representation is 0xC8*0xC8=0x9C40, which is 1 byte data multiplied by 1 byte data, and the result becomes 2 byte data, but since the final result needs to be 1 byte data, the high 9C of 0x9C40 is discarded, and only 0x40 is reserved;

[0022] ③The accumulation is used, and only the low byte is reserved, and the MK1, MK2,..., MK8 are collected to obtain the digest M;

[0023] ④For a single file in the file system, the digest value is obtained by using steps ①-③, and for a directory in the file system, a digest file is created, the names and digest values of all files contained in the directory are recorded by using the digest file, and the digest value of the digest file is used as the digest value of the entire directory.

[0024] According to the application, in step (3), the synchronization process is:

[0025] The file directory to be synchronized is added to the directory synchronization queue;

[0026] If the synchronization queue is empty, the synchronization process is ended, if the synchronization queue is not empty, a directory synchronization task is taken out from the queue, the digest file A of the file directory in the download server is downloaded, and the digest file B in the local directory is compared, the comparison content is a single data element, that is, the file / directory name and the digest value:

[0027] For a single data element, if the digest file A has it and the digest file B does not have it: it is a new item, if the item is a file, it can be directly downloaded, if the item is a directory, it is added to the directory synchronization queue and waits for processing;

[0028] For a single data element, if the digest file A does not have it and the digest file B has it: it is a deleted item, and the corresponding item in the terminal directory can be directly deleted;

[0029] For a single data element, if the element is in the summary file A and also in the summary file B, then the summary value of the element is compared, if the summary values are the same, it indicates that the element has not been changed, if the summary values are different, the element is a file, the file is downloaded again and replaced, the element is a directory, the directory is added to the directory synchronization queue and waits for processing;

[0030] If all the elements in the current directory summary file are processed, the current element is deleted from the synchronization queue, and the synchronization request in the synchronization queue is processed until the synchronization queue is empty, thus the synchronization task of the entire file directory is completed.

[0031] According to the application, preferably, in step (4), since the contents of the summary files in the summary tree-based directory are the same, the files in the two directories are guaranteed to be the same, thus during routine maintenance, it is only necessary to check whether the summary values of the summary files under the server synchronization directory are consistent with the summary values of the summary files under the local synchronization directory, so as to determine whether the files under the synchronization directory are updated, which greatly reduces the network and work expenses of routine maintenance.

[0032] The above calculation process is easy to be divided into small calculation units due to the independence of each other, which facilitates parallel processing using multi-core, and since the calculation mainly uses multiplication and addition, it is friendly to optimization using the vectorized instructions in various architectures.

[0033] A file system synchronization system based on a vectorizable summary algorithm, comprising:

[0034] A summary tree construction module, configured to construct a summary tree for a file directory to be synchronized using a summary algorithm on the constituent structure of the file directory;

[0035] An instruction sending module, configured to store the file directory with the generated summary tree and the subordinate files of the file directory on a remote server, and send a synchronization instruction to a terminal;

[0036] A synchronization module, configured to synchronize the file directory of the terminal;

[0037] A maintenance module, configured to perform routine maintenance on the synchronization directory.

[0038] The application has the following advantages:

[0039] The application defines a set of vectorizable summary algorithms, and uses the same to construct a summary tree, which is applied in the synchronization process of a file system. The summary algorithm is vectorizable, and the data is converted into a vector to calculate, so that the parallel processing capability of a modern computer can be fully utilized, and the speed of summary generation is significantly improved. Meanwhile, the content needing synchronization can be quickly identified and corresponding operation is performed. For a task of periodic file system synchronization in large quantities, the correctness and speed can be efficiently unified, and the efficiency of file system synchronization is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 An example diagram of the summary algorithm of the application.

[0041] Figure 2 A file system summary tree construction diagram of the application.

[0042] Figure 3 A summary tree logic diagram of the application.

[0043] Figure 4 A use schematic diagram of the directory summary file of the application in synchronization.

[0044] Figure 5 A file system directory synchronization flowchart based on the summary tree structure of the application. DETAILED DESCRIPTION

[0045] The application will be further described below by way of examples and in conjunction with the drawings, but is not limited thereto.

[0046] Example 1

[0047] The example provides a file system synchronization method based on a vectorizable summary algorithm, and the steps are as follows:

[0048] (1) For the file directory needing synchronization, a summary tree is constructed for the component structure of the file directory using a summary algorithm;

[0049] The summary tree construction process includes file directory processing and file processing in the directory, and the details are as follows:

[0050] File directory: The file directory contains a summary file, which is used to describe the summary value of the file and the next level directory in the file directory, wherein the summary value of the summary file is used as the summary value of the file directory;

[0051] File: The summary value of the file is calculated using the summary algorithm, and the name of the file and the calculated summary value are stored in the summary file of the directory where the file is located;

[0052] The steps of the summary algorithm are as follows:

[0053] ①First, select 8 linearly independent key vectors K1, K2...K8, and cut the original data into independent data blocks D1, D2...Dn in units of 8 bytes (the part less than 8 bytes at the end is padded with 0xFF, 0xFF is a 16-bit number, 1 byte is represented by 8 bits, 8 bits all 1 in binary representation is 11111111, converted to 16-bit hexadecimal is 0xFF);

[0054] ②Respectively, consider the independent data blocks D1, D2...Dn as vectors and do vector product with the key vector K1 to get F1K1, F2K1...FnK1, and collect the results of the vector product to get MK1 (accumulate, discard the high data in each dimension, only keep one byte), similarly, do vector product with the key vector K2, K3...K8 and collect the results to get MK2, MK3...MK8;

[0055] High data is the data overflowed when doing multiplication, such as 200*200=40000, using unsigned hexadecimal representation is 0xC8*0xC8=0x9C40, which is 1 byte data multiplied by 1 byte data, the result becomes 2 byte data, but since the final result needs to be 1 byte data, the high 9C of 0x9C40 is discarded, and the result is only saved as 0x40;

[0056] ③Use accumulation, only keep the low byte, and collect MK1, MK2...MK8 to get the digest M;

[0057] ④For a single file in the file system, use steps ①-③ to get the digest value, for a directory in the file system, create a digest file, use the digest file to record the names and digest values of all files it contains, and use the digest value of the digest file as the digest value of the entire directory.

[0058] Digest tree: a tree structure of digest values formed by combining the digest values of each subdirectory / file in the folder and the tree structure of the directory in the file system, see Figure 2 、 Figure 3 for comparison, Figure 2 that is, use the digest algorithm to calculate the digest values of each subdirectory / file, Figure 3 that is, the tree structure of the digest values formed by combining the tree structure of the directory, that is, the digest tree.

[0059] (2) Store the file directory and its subordinate files with the generated digest tree on the remote server, and send a synchronization instruction to the terminal;

[0060] (3) The terminal performs file directory synchronization, the synchronization process is:

[0061] Add the file directory that needs to be synchronized to the directory synchronization queue;

[0062] If the synchronization queue is empty, it means that the synchronization process is over, if the synchronization queue is not empty, a directory synchronization task is taken out of the queue, for the directory synchronization task, the summary file A of the file directory in the download server is downloaded and compared with the summary file B in the local directory, the comparison content is a single data element, i.e. file / directory name, summary value, as shown in Figure 4

[0063] For a single data element, if there is in the summary file A and there is not in the summary file B (such as file 1 in Figure 4 ), it means that the item is a new item, if the item is a file, it can be directly downloaded, if the item is a directory, it is added to the directory synchronization queue and waits for processing;

[0064] For a single data element, if there is not in the summary file A and there is in the summary file B (such as file 2 in Figure 4 ), it means that the item is a deleted item, the corresponding item in the terminal directory can be directly deleted;

[0065] For a single data element, if there is in the summary file A and there is also in the summary file B: the summary value of the item is compared, if the summary values are the same (such as file 3 in Figure 4 ), it means that the item is not changed, if the summary values are different (such as file 4 in Figure 4 ), when the item is a file, the item file is re-downloaded and replaced, when the item is a directory, the directory is added to the directory synchronization queue and waits for processing;

[0066] If all the items in the current directory summary file are processed, the current item is deleted from the synchronization queue, and the synchronization request in the synchronization queue is continuously processed until the synchronization queue is empty, thus the synchronization task processing of the entire file directory is completed.

[0067] (4) Daily maintenance of the synchronization directory, specifically, since the contents of the summary files in the summary tree-based directory are the same, it can be ensured that the files in the two directories are the same, therefore, when daily maintenance is performed, it is only necessary to check whether the summary values of the summary files under the server synchronization directory are consistent with the summary values of the summary files under the local synchronization directory, so as to determine whether the files under the synchronization directory are updated, which greatly reduces the network and work expenses of daily maintenance.

[0068] Embodiment 2

[0069] The embodiment provides a file system synchronization system based on a vectorizable summary algorithm, comprising:

[0070] a summary tree construction module, configured to use a summary algorithm to construct a summary tree for a file directory to be synchronized;

[0071] ​The instruction sending module is used for storing the file directory and the subordinate files of the generated summary tree on a remote server and sending a synchronization instruction to the terminal;

[0072] The synchronization module is used for synchronizing the file directory of the terminal.

[0073] The maintenance module is used for daily maintenance of the synchronized directory.

[0074] The above description of the specific embodiments of the present application in combination with the accompanying drawings is not a limitation on the protection scope of the present application, and those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A file system synchronization method based on a vectorizable summarization algorithm, characterized in that, The steps are as follows: (1) For the file directory to be synchronized, use the digest algorithm to construct a digest tree for the composition structure of the file directory; The digest algorithm steps are as follows: ① First, select eight linearly independent secret key vectors K1, K2, …, K8, and divide the original data into independent data blocks D1, D2, …, Dn with 8 bytes as a unit; ② Respectively, consider the independent data blocks D1, D2, …, Dn as vectors and do vector multiplication with the secret key vectors K1 to obtain F1K1, F2K1, …, FnK1, and collect the results of the vector multiplication to obtain MK1. Similarly, do vector multiplication with the secret key vectors K2, K3, …, K8 and collect the results to obtain MK2, MK3, …, MK8; ③ Use accumulation and only keep the low bytes, and collect MK1, MK2, …, MK8 to obtain the digest M; ④ For a single file in the file system, use steps ①-③ to obtain the digest value. For a directory in the file system, create a digest file, use the digest file to record the names and digest values of all files it contains, and use the digest value of the digest file as the digest value of the entire directory; (2) Store the file directory with the generated digest tree and its subordinate files on a remote server, and send a synchronization instruction to the terminal; (3) The terminal performs file directory synchronization; (4) Daily maintenance of the synchronized directory.

2. The file system synchronization method based on vectorizable digest algorithm as claimed in claim 1, wherein, In step (1), the digest tree construction process includes file directory processing and file processing in the directory, which is as follows: File directory: The file directory contains a digest file, which describes the digest values of the files and the next level of directories in the file directory. The digest value of the digest file is used as the digest value of the file directory; File: Use the digest algorithm to calculate the digest value of the file, and store the name of the file and the calculated digest value in the digest file in the directory where the file is located; Digest tree: The digest values of the subdirectories / files in the folder are combined with the tree structure of the directories in the file system to form a tree structure of digest values.

3. The file system synchronization method based on vectorizable digest algorithm as claimed in claim 2, wherein, In step (3), the synchronization process is as follows: Add the file directory that needs to perform synchronization to the directory synchronization queue; If the synchronization queue is empty, the synchronization process is complete. If the synchronization queue is not empty, take a directory synchronization task from the queue. For this directory synchronization task, download the digest file A of the file directory from the server and compare it with the digest file B in the local directory. The comparison content is a single data element, i.e., the file / directory name and the digest value: For a single data element, if it is in digest file A but not in digest file B: it is a new item. If the item is a file, it can be directly downloaded. If the item is a directory, it is added to the directory synchronization queue for processing; For a single data element, if it is not in digest file A but in digest file B: it is a deleted item, which can be directly deleted from the terminal directory; For a single data element, if it is in both digest file A and digest file B: compare the digest values. If the digest values are the same, the item has not been changed. If the digest values are different, the item is a file, which is downloaded again and replaced. The item is a directory, which is added to the directory synchronization queue for processing; If all items in the current directory digest file are processed, the current item is deleted from the synchronization queue, and the synchronization request in the synchronization queue is continuously processed until the synchronization queue is empty, and thus the synchronization task processing of the entire file directory is completed.

4. The file system synchronization method based on vectorizable digest algorithm as claimed in claim 3, wherein, In step (4), specifically, since the contents of the digest files in the digest tree-based directory are the same, it can be ensured that the files in the two directories are the same, and thus during routine maintenance, it is only necessary to check whether the digest values of the digest files under the server synchronization directory are consistent with the digest values of the digest files under the local synchronization directory, so as to determine whether the files under the synchronization directory are updated.

5. A file system synchronization system based on a vectorizable summarization algorithm, characterized in that, The file system synchronization method based on the vectorable digest algorithm in claim 1 comprises: a digest tree construction module, configured to use a digest algorithm to construct a digest tree for the file directory to be synchronized; an instruction sending module, configured to store the file directory and the subordinate files of the file directory with the generated digest tree on a remote server, and send a synchronization instruction to a terminal; a synchronization module, configured to synchronize the file directory for the terminal; a maintenance module, configured to perform routine maintenance on the synchronization directory.

Images