A data processing method and apparatus

Abstract

The application discloses a data processing method and device, wherein the data processing method comprises the following steps: obtaining a first data structure of a to-be-stored file and a second data structure of a stored target file; determining difference data according to node storage information of the first data structure and the second data structure; the difference data is data different from the target file in the to-be-stored file; and storing the difference data.

Description

Technical Field

[0001] This application belongs to the field of data storage technology, and in particular relates to a data processing method and apparatus. Background Technology

[0002] When storing data based on known storage technologies, such as using third-party cloud storage media to store user document data, there are often problems such as high storage space consumption and waste of storage resources. How to solve this problem has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] A data processing method, comprising:

[0005] Obtain the first data structure of the file to be stored, and obtain the second data structure of the already stored target file;

[0006] The difference data is determined based on the storage information of each node in the first data structure and the second data structure; the difference data is the data in the file to be stored that is different from the target file.

[0007] The difference data is stored.

[0008] Optionally, obtaining the first data structure of the file to be stored and the second data structure of the already stored target file includes:

[0009] Obtain the first metadata tree of the file to be stored and the second metadata tree of the target file;

[0010] The file's metadata tree is a tree-like data structure built by dividing the file into at least one fragment according to a set fragment size. It is used to store file fragment information. A leaf node in the metadata tree corresponds to a fragment of the file. The leaf node is used to store at least the summary information of the corresponding fragment, and the non-leaf nodes are used to store the summary information of the merged result of the summary information in the corresponding child nodes.

[0011] Optionally, obtaining the first metadata tree of the file to be stored includes:

[0012] Construct the first metadata tree of the file to be stored;

[0013] Obtaining the second metadata tree of the target file includes at least one of the following methods:

[0014] Construct the second metadata tree of the target file;

[0015] Read the second metadata tree of the target file from the preset storage.

[0016] Optionally, the difference data is determined based on the storage information of each node in the first data structure and the second data structure, including:

[0017] Compare the summary information in the nodes of the first metadata tree and the second metadata tree;

[0018] Based on the comparison results, a first target leaf node is identified in the first metadata tree whose digest information is the same as that stored in the corresponding leaf node of the second metadata tree;

[0019] The data content of the leaf nodes other than the first target leaf node in the first metadata tree corresponding to the fragments in the file to be stored is determined as the difference data.

[0020] Optionally, comparing the digest information in the nodes of the first metadata tree and the second metadata tree, and determining a first target leaf node in the first metadata tree whose digest information is the same as that stored in the corresponding leaf node of the second metadata tree based on the comparison result, includes:

[0021] The summary information in the peer nodes of the first metadata tree and the second metadata tree is compared in a top-down manner; the number of subtree levels corresponding to the peer nodes is the same.

[0022] Based on the comparison results, a first target node is identified that stores the same digest information as the corresponding node in the second metadata tree in the first metadata tree;

[0023] Each leaf node contained in the first target node is determined as the first target leaf node.

[0024] Optionally, the leaf nodes in the file metadata tree are also used to store a first location index of the storage location of the fragment corresponding to the leaf node; after storing the difference data, the method further includes:

[0025] Obtain the first location index of the storage location of the difference data, and store the obtained first location index in the leaf node corresponding to the difference data in the first metadata tree;

[0026] Obtain the first position index of the first target leaf node in the second target leaf node corresponding to the first target leaf node in the second metadata tree, and store the obtained first position index in the first target leaf node; the second target leaf node is a leaf node in the second metadata tree with the same summary information as the first target leaf node.

[0027] Optionally, the method further includes:

[0028] In response to a file access request, the system obtains the first position index in each leaf node of the target metadata tree, and obtains the target fragments at each position indicated by each first position index; the target metadata tree is the metadata tree indicated by the file access request.

[0029] Merge each of the target fragments to obtain the corresponding merged result file;

[0030] The merged result file is sent back to the requesting end.

[0031] Optionally, the leaf nodes in the file metadata tree are also used to store the second position index of the content position of the corresponding fragment in its file; the merging of the target fragments includes:

[0032] Obtain the second position index corresponding to each target fragment from each leaf node of the target metadata tree;

[0033] The target fragments are merged according to their respective second position indices.

[0034] Optionally, before obtaining the second data structure of the stored target file, the method further includes:

[0035] Determine a first file from the existing stored files that has the same structural layout as the file to be stored;

[0036] The target file is determined from the first file that meets similar conditions to the content of the file to be stored.

[0037] A data processing apparatus, comprising:

[0038] The acquisition module is used to obtain the first data structure of the file to be stored and the second data structure of the already stored target file;

[0039] The determining module is used to determine the difference data based on the storage information of each node in the first data structure and the second data structure; the difference data is the data in the file to be stored that is different from the target file.

[0040] A storage module is used to store the difference data.

[0041] As can be seen from the above scheme, this application discloses a data processing method and apparatus, wherein the data processing method includes: obtaining a first data structure of a file to be stored, and obtaining a second data structure of a target file that has been stored; determining difference data based on the storage information of each node of the first data structure and the second data structure; the difference data being data in the file to be stored that is different from the target file; and storing the difference data. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0044] Figure 2 is an example of a metadata tree provided in an embodiment of this application;

[0045] Figure 3 This application provides an example of storing difference data by comparing metadata tree nodes;

[0046] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0047] Figure 5 This is an example of storing file data content on a storage medium at the granularity of data fragmentation, provided as an embodiment of this application.

[0048] Figure 6 This is an example of storing file data content at the file level in known technologies;

[0049] Figure 7 Another flowchart illustrating the data processing method provided in this application embodiment;

[0050] Figure 8 This is a structural diagram of the data processing apparatus provided in the embodiments of this application;

[0051] Figure 9 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] This application discloses a data processing method and apparatus to address the problems of high storage space consumption and wasted storage resources in existing storage technologies. The disclosed data processing method and apparatus can be used in electronic devices in various general-purpose or special-purpose computing environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, etc.

[0054] See Figure 1 The data processing method flowchart shown in this application discloses at least the following data processing methods:

[0055] Step 101: Obtain the first data structure of the file to be stored, and obtain the second data structure of the already stored target file.

[0056] The files to be stored and the files already stored can be, but are not limited to, any type of data file such as images, audio / video, news, documents, or various office documents, without any restrictions, depending on the actual needs.

[0057] The target file can be any file in the existing files, such as a file traversed from the existing files using a traversal method, or a similar file selected from the existing files using a corresponding filtering method that has similarity to the content of the file to be stored. There are no restrictions.

[0058] The first data structure of the file to be stored and the second data structure of the already stored target file can both be tree-like data structures, which are referred to as the file's metadata tree in this embodiment. Of course, other structures with directional relationships between nodes are also applicable, in addition to tree-like data structures. In this step, the first metadata tree of the file to be stored and the second metadata tree of the already stored target file can be obtained accordingly.

[0059] The file's metadata tree is a tree-like data structure used to store file fragment information, constructed by dividing the file into at least one fragment according to a set fragment size. File fragment information may include, but is not limited to, any one or more of the following: fragment summary information, summary information of the merged result of different fragment summary information, location index of fragment storage location, location index of the content location corresponding to the fragment in the file, fragment size, etc.

[0060] A leaf node in the file metadata tree corresponds to a fragment of the file. Optionally, the leaf node is used to store at least the summary information of the corresponding fragment, and the non-leaf nodes are used to store the summary information of the merged result of the summary information in the corresponding child nodes.

[0061] The summary information of a shard can be a hash value obtained by hashing the data content of the shard using a preset hash algorithm. The summary information of the merged summary results of the child nodes of a non-leaf node can be a hash value obtained by merging the hash values ​​of the child nodes of the non-leaf node and further hashing the merged result using the preset hash algorithm. Optionally, the merged result can be the result of concatenating the binary strings of the hash values ​​of the child nodes of the non-leaf node, or it can be the result of performing logical operations on the hash values ​​of the child nodes.

[0062] The metadata tree of a file can be a binary tree or a multi-way tree, preferably a binary tree. This application embodiment will mainly use a binary tree as an example for illustration. For example, it can be a binary tree with predefined node information types to meet the data storage and processing requirements of this application embodiment.

[0063] In the binary tree data structure, optionally, each non-leaf node in the file metadata tree contains two child nodes. For cases where a certain non-leaf node actually corresponds to only one child node, such as when the number of file fragments is odd, a certain non-leaf node corresponds to only one child node. See the metadata tree example provided in Figure 2(a). The number of child nodes of the non-leaf node can be increased to 2 by repeating / copying the existing child nodes of the non-leaf node. It is easy to understand that in this case, the summary information in the non-leaf node is specifically the summary information of the result of merging the identical summary information in the two identical child nodes it contains.

[0064] Of course, for cases where one or more non-leaf nodes actually correspond to only one child node, it is not necessary to padded them to two child nodes. The metadata tree can be constructed according to the actual number of child nodes it contains. See Figure 2(b), which provides an example of a metadata tree in this case.

[0065] Step 102: Determine the difference data based on the storage information of each node in the first data structure and the second data structure; the difference data is the data in the file to be stored that is different from the target file.

[0066] In this step, the summary information in the nodes of the first metadata tree and the second metadata tree can be compared. Based on the comparison result, the first target leaf node in the first metadata tree that stores the same summary information as the corresponding leaf node in the second metadata tree is determined. The data content of the corresponding fragments in the file to be stored for the leaf nodes other than the first target leaf node in the first metadata tree is determined as the difference data.

[0067] For comparing the summary information in the nodes of the two metadata trees, optionally, each node in the first metadata tree and the second metadata tree can be traversed in a top-down manner to generate a node pair consisting of nodes in the two metadata trees, and the summary information stored in the two nodes in the node pair can be compared to determine the first target leaf node whose summary information is the same as that stored in the corresponding leaf node of the second metadata tree.

[0068] To further improve comparison efficiency, this application further proposes the concept of peer layer between two metadata trees. Specifically, the peer layer refers to the layer corresponding to the nodes of the subtrees contained in the two metadata trees with the same number of layers. The subtrees corresponding to the peer layer nodes of the two metadata trees all have the same number of layers.

[0069] Taking the first metadata tree of the file to be stored and the second metadata tree of the target file as an example, assuming that the first metadata tree has two layers, from top to bottom: L10 and L11, and the second metadata tree has three layers, from top to bottom: L20, L21 and L22, where the number of subtree layers contained in the nodes of the first metadata tree layer L10 and the second metadata tree layer L21 is 1, then layer L10 and layer L21 can be regarded as an equivalent layer of the first metadata tree and the second metadata tree.

[0070] Based on the proposed peer layer, when comparing nodes of two metadata trees (first metadata tree and second metadata tree) to determine the difference data, the summary information in the peer layer nodes between the first metadata tree of the file to be stored and the second metadata tree of the target file can be compared in a top-down manner. Based on the comparison results, the first target node in the first metadata tree that stores the same summary information as the corresponding node in the second metadata tree is determined, and each leaf node contained in the first target node is determined as the first target leaf node in the first metadata tree that stores the same summary information as the corresponding leaf node in the second metadata tree.

[0071] Based on the determination of the first target leaf node, the data content of the leaf nodes other than the first target leaf node in the first metadata tree corresponding to the fragments in the file to be stored is determined as the difference data between the file to be stored and the target file.

[0072] By comparing the summary information in the peer nodes between the first metadata tree and the second metadata tree, it is not necessary to compare the summary information of all nodes between the two metadata trees. Instead, it only compares the nodes between the two metadata trees that may contain the same summary information. This avoids the situation where the comparison results fail due to structural inconsistencies, reduces the workload of the comparison process, and improves efficiency.

[0073] In the process of comparing the summary information of peer-level nodes of the first metadata tree and the second metadata tree, optionally, each node in the peer-level of the first metadata tree can be compared with each node in the peer-level of the second metadata tree. During the comparison, if it is determined that the summary information of two nodes in the peer-level of the two metadata trees is the same, then in subsequent comparisons at that peer-level, the comparison of these two nodes can be skipped. That is, only the nodes with different stored summary information are compared at the next lower level until the comparison is complete. For example, assuming that the first metadata tree includes two nodes, Node1 and Node2, in peer-level Lx, and the second metadata tree includes two nodes, Node3 and Node4, then Node1 can be compared with Node3 and Node4 respectively, and Node2 can be compared with Node3 and Node4 respectively. If, during the comparison process, Node1 is first compared with Node3 and Node4 respectively, and the summary information of Node1 and Node4 is the same, then in subsequent comparisons, Node1 and Node4 can be skipped, and only Node2 and Node3 need to be compared.

[0074] The determined difference data specifically includes the data content corresponding to one or more fragments of the file to be stored.

[0075] It should be noted that the first target node may be empty or not empty, and it can be a node at any level in the comparison tree, such as the root node, intermediate level nodes, leaf nodes, etc. The first target leaf node may also be empty or not empty. If the first target leaf node is empty, it indicates that there are no identical data fragments between the file to be stored and the target file; all fragments of the file to be stored contain data that differs from the target file. If the first target leaf node is not empty, it indicates that there are identical data fragments between the file to be stored and the target file; the data content of fragments other than these identical fragments constitutes the data that differs from the target file.

[0076] If all leaf nodes of the first metadata are the first target leaf nodes, it indicates that all fragment data content of the file to be stored and the target file are the same, which in turn indicates that the overall file data content of the file to be stored and the target file are the same.

[0077] Step 103: Store the difference data.

[0078] Based on the determination of the differences between the file to be stored and the target file, this embodiment of the application only stores the differences, while the non-difference data (i.e. the same data) between the file to be stored and the target file are not stored again.

[0079] In summary, the data processing method disclosed in this application obtains a first data structure of the file to be stored and a second data structure of the already stored target file, determines the difference data (the difference data is the data in the file to be stored that is different from the target file) based on the storage information of each node in the first and second data structures, and stores the difference data. This method can effectively identify the difference data between the file to be stored and the already stored target file and selectively store only the difference data, thereby avoiding the repeated storage of non-difference data, saving system storage space, improving storage resource utilization, and avoiding storage resource waste.

[0080] like Figure 3 In the example, if data shards A and B in the document to be stored are determined to be the same as data shards C and D in the document already stored by comparing nodes in the metadata tree, then only the other shards of the document to be stored, excluding data shards A and B, need to be stored. Data shards A and B do not need to be stored repeatedly, thus saving storage space.

[0081] In one optional embodiment, when a file to be stored is obtained and needs to be stored, a first metadata tree of the file to be stored can be constructed in real time, thereby realizing the acquisition of the first metadata tree of the file to be stored.

[0082] Specifically, the file to be stored can be divided into at least one fragment according to the set fragment size, the digest information of each fragment can be determined (such as using a preset hash algorithm to calculate the data content of each fragment to obtain the corresponding hash value), and multiple leaf nodes can be created in a number equal to the number of fragments, each corresponding to one fragment. Each leaf node stores at least the digest information of the corresponding fragment. Then, non-leaf nodes are built layer by layer in a bottom-up manner until the root node is completed.

[0083] Taking a binary tree as an example, each non-leaf node can store the summary information of the merged summary information of its two child nodes. In cases where a certain non-leaf node actually corresponds to only one child node due to reasons such as an odd number of file fragments, the number of child nodes of that non-leaf node can be increased to two by repeating / copying its existing child nodes. This allows the summary information of the non-leaf node to be determined until the root node is constructed. After the root node is constructed, the first metadata tree corresponding to the file to be stored is obtained.

[0084] The set fragment size can be, but is not limited to, 128K or 512K.

[0085] In an optional implementation, for a pre-stored second metadata tree of a target file, the pre-stored second metadata tree of the target file can be read directly from a preset storage.

[0086] The pre-stored second metadata tree is specifically a metadata tree that is synchronously built and stored for the target file when the target file is stored and processed in historical time. Its leaf nodes correspond one-to-one with the fragments of the target file and at least store the summary information of the corresponding fragment data content. The non-leaf nodes store the summary information of the merged result of the summary information of the child nodes.

[0087] However, this is not the only possibility. The second metadata tree of the target file can also be constructed in real-time when the storage processing of the file requires obtaining the second metadata tree of the already stored target file. Specifically, the various stored fragments contained in the target file can be read, the digest information of each stored fragment can be determined (such as calculating the hash value of the data content of each fragment using a preset hash algorithm), and the nodes of each level used to form the second metadata tree of the target file can be constructed sequentially from bottom to top until the root node is completed, thus obtaining the second metadata tree of the target file.

[0088] In the implementation of real-time construction of the second metadata tree, the summary information of each segment of the target file can also be stored synchronously when the target file is stored in historical time. This allows the summary information of each segment of the target file to be directly obtained for the rapid real-time construction of the second metadata tree when the second metadata tree of the stored target file is required for storage processing.

[0089] This embodiment obtains the first metadata tree of the file to be stored and the second metadata tree of the target file by utilizing the above-described methods. This facilitates the subsequent determination and storage of the difference data between the file to be stored and the already stored target file, based on the first and second metadata trees, thus avoiding the repeated storage of non-difference data.

[0090] In an optional embodiment, for the case of selecting similar files with content similar to the file to be stored as target files from the stored files, the selection of target files can be achieved through any one or more combinations of the following methods:

[0091] Method 1: Determine a file with the same structure and layout as the file to be stored from the existing stored files.

[0092] The structural layout of a file may include, but is not limited to, the structural layout characterized by a file template and / or the file's data organization format.

[0093] In implementing the technical solution disclosed herein, the applicant discovered that files with the same structural layout are more likely to have similar content than files with different structural layouts. For example, archives or academic papers have specific templates / formats, resulting in these files having the same structural layout. For archive files to be stored, it is easier to generate similar files with overlapping content among historical archives with the same structural layout. Similarly, for academic papers to be stored, it is easier to generate similar files with overlapping content among historical academic papers with the same structural layout. This can improve the hit rate of identical data when comparing metadata tree nodes of files to be stored with those of already stored files, thereby improving the utilization rate of storage resources. Based on this, Method 1 identifies files with the same structural layout as the file to be stored from the already stored files and uses these files as target files to participate in the metadata tree comparison process with the file to be stored.

[0094] Files whose structure and layout differ from the file to be stored are considered non-target files and are not included in the metadata tree comparison process between the stored file and the file to be stored.

[0095] Method 2: Determine files from the existing files whose filenames meet the set conditions for similarity with the file to be stored.

[0096] If different files have similar filenames, there is a potential for overlapping file data, making it easier to generate files with overlapping content, which in turn helps reduce the demand for storage resources. Based on this, Method Two identifies files whose filenames meet the set similarity criteria to the file to be stored from the already stored files, and uses these as target files.

[0097] The similarity criteria here can be set as follows, but are not limited to: the semantic similarity of the file names of different files reaches a similarity threshold, or the file names of different files contain a set number of identical keywords.

[0098] For example, suppose the file to be stored and a certain stored file are only different in name, such as the file to be stored is named xxx.v2 and the stored file is named xxx.v1, the only difference being the version number v2 and v1. Then the stored file xxx.v1 can be identified as the target file.

[0099] Method 3: Determine files from the existing files that have the same category and / or domain as the file to be stored.

[0100] Compared to different categories or domains, files with overlapping content are more likely to be generated from the same category or domain. Based on this, Method 3 identifies files from the stored files that have the same category and / or domain as the file to be stored, and uses them as target files to participate in the metadata tree comparison process between the target file and the file to be stored.

[0101] Method 4: Determine files from the stored files whose content titles and / or keywords match the set similarity criteria of the file content titles and / or keywords of the file to be stored.

[0102] The similarity criteria here can be set as follows, but are not limited to: the semantic similarity between the titles and / or keywords (such as the title / keywords of a paper) of different documents reaches a similarity threshold, or there is a set number of identical keywords in the titles / keywords of different documents.

[0103] This method uses the content title and / or keywords of a file as a basis to identify files whose content title and / or keywords match the content title and / or keywords of the file to be stored from the stored files. These files are then used as target files to participate in the metadata tree comparison process between the target file and the file to be stored.

[0104] The following provides a preferred implementation for filtering target files from stored files. This implementation includes a two-stage filtering process, in which a first stage identifies a first file from the stored files that has the same structural layout as the file to be stored, and then in a second stage, a target file from the first file that has content that satisfies similarity conditions to the file to be stored.

[0105] The similarity criteria here can be any one of the similarity criteria set in methods two through four above, or a combination of different similarity criteria in methods two through four. Through the above two stages of screening, files with high overlap or high probability of content overlap with the file to be stored are selected as target files as efficiently and quickly as possible, so that they can participate in the metadata tree comparison processing between the target file and the file to be stored.

[0106] This embodiment utilizes a corresponding filtering method to select similar files from the stored files that have content similarity to the file to be stored as target files. This avoids performing metadata tree comparison processing on all stored files and the file to be stored (to determine the difference data), effectively reducing the workload of storing the file to be stored and improving processing efficiency.

[0107] In an optional embodiment, the leaf nodes in the file metadata tree are further used to store a first location index of the storage location of the corresponding fragment, so as to access the required fragment data content based on the corresponding first location index. The first location index of the fragment storage location can specifically be the physical location information or logical location information corresponding to the fragment storage location, without limitation.

[0108] Accordingly, see Figure 4 The data processing method disclosed in this application, after storing the difference data, may further include:

[0109] Step 104: Obtain the first location index of the storage location of the difference data, and store the obtained first location index in the leaf node corresponding to the difference data in the first metadata tree.

[0110] The difference data includes one or more fragments of the file to be stored that differ from the fragments of the target file. In this embodiment, the fragments corresponding to the difference data are referred to as difference fragments.

[0111] The storage of differential data refers to the storage of one or more differential fragments corresponding to the differential data in the file to be stored.

[0112] In this step, specifically, based on the actual storage location of each difference fragment in the storage medium when storing difference data, the first location index of the storage location of each difference fragment can be obtained, and the obtained first location index can be stored in the leaf node corresponding to the first metadata tree of the file to be stored for that difference fragment.

[0113] Step 105: Obtain the first position index of the second target leaf node in the second metadata tree corresponding to the first target leaf node, and store the obtained first position index in the first target leaf node; the second target leaf node is a leaf node in the second metadata tree that has the same summary information as the first target leaf node.

[0114] The data other than the difference data in the file to be stored refers to the fragments other than the difference fragments in each fragment of the file to be stored. In this embodiment, these are referred to as non-difference fragments (indicating that the fragment in the file to be stored has a corresponding fragment in the target file that is the same as it).

[0115] A non-differentiated fragment is specifically the fragment corresponding to the first target leaf node in the first metadata tree in the file to be stored. For a non-differentiated fragment in the file to be stored, since the same fragment already exists in the target file and has been stored, the non-differentiated fragment is not stored repeatedly when the file to be stored is processed (so that the same fragment between different files is stored only once). The reference relationship of the stored data content of the non-differentiated fragment can be directly constructed in the first target leaf node corresponding to the non-differentiated fragment in the first metadata tree. Specifically, the first position index of the non-differentiated fragment in the second target leaf node corresponding to the second metadata tree of the target file can be obtained directly from the second metadata tree of the target file, and the obtained first position index is stored in the first target leaf node corresponding to the non-differentiated fragment in the first metadata tree of the file to be stored.

[0116] It is easy to understand that the first target leaf node corresponding to the non-differential shard in the first metadata tree and the second target leaf node corresponding to the second metadata tree of the target file respectively store the same summary information, which is the summary information of the non-differential shard.

[0117] Optionally, duplicate fragments within the same file can be identified so that only one copy of each duplicate fragment is stored. Specifically, the digest information of nodes at the same level within the same file can be compared. If two nodes at the same level within the same file, such as Node1 and Node2, store the same digest information, it is considered that the fragments corresponding to the leaf nodes in the subtrees contained by these two nodes (Node1 and Node2) are the same. Therefore, only the fragments corresponding to the leaf nodes in the subtrees contained by Node1 and Node2 need to be stored. That is, only the fragments corresponding to the leaf nodes in the subtree contained by Nodex (Nodex is either Node1 or Node2) are stored. A reference relationship for the stored duplicate fragment is established in the leaf nodes corresponding to Nodey (other than Nodex) in Node1 and Node2. For example, the location index of the data content of the duplicate fragment in the storage medium can be directly written into the leaf nodes corresponding to Nodey.

[0118] It should be noted that if the identified duplicate fragment is a non-differential fragment relative to the already stored target file, that is, the data content of the duplicate fragment is already stored in the storage medium, then the reference relationship of the data content of the duplicate fragment in the storage medium can be directly constructed in each leaf node corresponding to the duplicate fragment (such as the leaf nodes corresponding to Node1 and Node2 above). For example, the position index of the data content of the duplicate fragment in the storage medium can be directly written into each leaf node corresponding to the duplicate fragment.

[0119] Optionally, in practical applications, the number of times each stored shard is referenced can also be recorded and maintained.

[0120] This embodiment stores the first location index of the storage location of the corresponding fragment in the leaf node of the file metadata tree, which facilitates quick indexing of each fragment of the file based on the file metadata tree, so as to support access to the file data content.

[0121] Based on the method of this application embodiment, the files stored in the storage medium are essentially data fragments, that is, the data content of the files is stored in the storage medium at the granularity of data fragments. For details, please refer to... Figure 5 For example, for the same fragment in different files, only one copy is stored to avoid duplicate storage, which differs from known technologies that store file data content at the file level (such as...). Figure 6 Compared to the example of [previous method], this can significantly reduce the amount of storage space required and improve the utilization rate of storage resources.

[0122] In an alternative embodiment, such as Figure 7 As shown, the data processing method disclosed in the embodiments of this application may further include the following processing:

[0123] Step 701: In response to the file access request, obtain the first position index in each leaf node of the target metadata tree, and obtain the target fragments at the respective positions indicated by each first position index.

[0124] The target metadata tree is the metadata tree indicated by the file access request. The file access request may carry the file identifier of the file to be accessed and / or the metadata tree identifier corresponding to the file to be accessed.

[0125] When a file access request carries a metadata tree identifier, the target metadata tree can be directly determined based on the carried metadata tree identifier. When a file access request does not carry a metadata tree identifier, the target metadata tree identifier corresponding to the file access request can be determined based on the file identifier carried in the file access request and the pre-built correspondence between file identifiers and metadata tree identifiers. Then, based on the target metadata tree identifier, the target metadata tree indicated by the file access request can be determined.

[0126] After determining the target metadata tree indicated by the file access request, each leaf node of the target metadata tree is accessed, the first position index stored in each leaf node of the target metadata tree is read, and based on the read first position index, each target fragment at the position indicated by each first position index is obtained.

[0127] Step 702: Merge each of the target fragments to obtain the corresponding merged result file.

[0128] After obtaining each target fragment, the target fragments are merged. Specifically, according to the relative positions of the target fragments, the target fragments are spliced ​​into a complete data file to obtain the corresponding merged result file.

[0129] Optionally, leaf nodes in the file metadata tree can also store a second position index of the content location of the fragment corresponding to the leaf node within its parent file. This second position index can be, but is not limited to, the order or number of the fragment corresponding to the leaf node within its parent file. It's easy to understand that the file to which the fragment corresponding to the leaf node belongs is the file corresponding to the metadata tree containing the leaf node.

[0130] Accordingly, when merging the target fragments in this step, the second position index corresponding to each target fragment can be obtained from each leaf node of the target metadata tree, and the target fragments can be merged according to the second position index corresponding to each target fragment. For example, the order corresponding to each target fragment can be obtained from each leaf node of the target metadata tree, and the target fragments can be merged into a complete data file in order to ensure that the relative positions of each target fragment in the merged file are correct.

[0131] Step 703: Send the merged result file back to the requesting end.

[0132] Finally, the merged result file can be sent back to the requesting end, enabling the requesting end to read and access the required data file.

[0133] It is worth noting that the execution order of steps 701-703 and steps 101-103 is not limited and can be determined according to the actual file storage / access needs.

[0134] For the accessed file, the requesting user can also perform modification or deletion operations on the file as needed.

[0135] If a user modifies a file, they can either upload the modified file (e.g., to the cloud) or save the updated file, triggering the storage service (e.g., the cloud) to process the updated file. The storage service receives the updated file, divides it into segments according to the set segmentation strategy and size, and constructs a corresponding metadata tree for each segment.

[0136] Optionally, the storage service provider can compare the metadata tree corresponding to the updated file with the metadata tree corresponding to the original file before modification to determine which fragments have changed (added fragments, deleted fragments, modified fragments) compared to the original file, and perform corresponding storage processing on the changed fragments. For ease of description, the metadata tree corresponding to the updated file is called the updated metadata tree, and the metadata tree corresponding to the original file of the updated file is called the original metadata tree.

[0137] For newly added / modified shards, the process compares the nodes in the updated metadata tree of the updated file with those in the metadata tree of the already stored target file to determine whether the newly added / modified shard is already stored or not. If it is already stored, the location index of the newly added / modified shard's storage location can be directly saved to its corresponding leaf node in the updated metadata tree, enabling a reference to the already stored data content of the shard. This avoids duplicate storage of the newly added / modified shard. If the storage service maintains a reference count for each already stored shard, the reference count for the newly added / modified shard can be updated synchronously. If the newly added / modified shard is not yet stored, its data content can be stored, and its location index can be saved to its corresponding leaf node in the updated metadata tree.

[0138] For the original fragment corresponding to the modified fragment in the original file, it is either deleted or retained depending on its reference status. If the original fragment corresponding to the modified fragment is not used by other files (i.e., it only belongs to the original file corresponding to the updated file), the original fragment can be deleted from the storage medium; otherwise, it is retained.

[0139] Similarly, for deleted fragments (fragments deleted from the original file compared to the updated file), processing is based on their actual reference status on the storage service side. If the deleted fragment only appears in the original file corresponding to the updated file, its data content is deleted on the storage service side in sync with the deletion operation of that fragment in the original file on the requesting end, so that its data content is no longer stored. If the deleted fragment is used by other files besides the original file corresponding to the updated file, its data content is maintained on the storage service side. Similarly, if the storage service side maintains the reference count for each stored fragment, the reference count corresponding to the deleted fragment can be updated synchronously.

[0140] After completing the storage processing of the changed shards (added shards, deleted shards, modified shards), the original metadata tree of the original file corresponding to the updated file is deleted, and the updated metadata tree corresponding to the updated file is stored, thereby realizing the update processing of the accessed file.

[0141] If a user requests and deletes a file, the storage service can query the file's metadata tree and process each data segment. For each data segment, if it is used only by that file, its data content is directly deleted, preventing it from being stored again, and the reference counting mechanism set for it is cleared. If the segment is used by other files (e.g., referenced by other files), its reference count can be maintained as needed (e.g., decrementing the reference count). After processing each file segment, the entire metadata tree of the file is deleted, thus completing the file deletion process on the storage service side.

[0142] The following is an application example of this application.

[0143] See also Figure 5 In this example, cloud storage resources are used for document data storage, and a document management server is set up for storing document data in the cloud. The server includes a document slicer for slicing documents and a data slice storage for storing the slices.

[0144] The workflows of the document fragmenter and the data fragment storage are as follows:

[0145] I. Document Splitter

[0146] 11) Segment the document to be stored;

[0147] For example, the document to be stored can be split into segments of 128K or 512K size to obtain segments of 128K or 512K size, with the size of the last segment depending on the actual situation.

[0148] 12) Based on the obtained fragments, construct the data structure corresponding to the document to be stored;

[0149] 13) Compare the data structures of the stored target documents to find the same data fragments and different data fragments between the document to be stored and the stored target documents;

[0150] 14) Record the location index of the storage location of the same data fragment and the reference count information of the same data fragment.

[0151] II. Data Slice Memory

[0152] 21) Obtain the data structure constructed by the document slicer for the document to be stored;

[0153] 22) Store the data that needs to be stored in cloud storage media in fragmented form (differential data fragmentation);

[0154] 23) Write the location index of the corresponding fragment in the storage medium to the leaf node of the data structure, and increment the reference count of the same data fragment. At the same time, a counting mechanism can be set for different data fragments to count the number of times they are referenced.

[0155] Corresponding to the data processing method described above, this application also provides a data processing apparatus, see below. Figure 8 The device includes:

[0156] The acquisition module 801 is used to obtain the first data structure of the file to be stored and the second data structure of the already stored target file;

[0157] The determining module 802 is used to determine the difference data based on the storage information of each node in the first data structure and the second data structure; the difference data is the data in the file to be stored that is different from the target file.

[0158] Storage module 803 is used to store the difference data.

[0159] In one embodiment, the acquisition module 801 is specifically used to: obtain the first metadata tree of the file to be stored and the second metadata tree of the target file;

[0160] The file's metadata tree is a tree-like data structure built by dividing the file into at least one fragment according to a set fragment size. It is used to store file fragment information. A leaf node in the metadata tree corresponds to a fragment of the file. The leaf node is used to store at least the summary information of the corresponding fragment, and the non-leaf nodes are used to store the summary information of the merged result of the summary information in the corresponding child nodes.

[0161] In one embodiment, when the acquisition module 801 obtains the first metadata tree of the file to be stored, it is specifically used to: construct the first metadata tree of the file to be stored;

[0162] When obtaining the second metadata tree of the target file, the acquisition module 801 is specifically used to perform one of the following methods: constructing the second metadata tree of the target file; or reading the second metadata tree of the target file from a preset storage.

[0163] In one embodiment, the determining module 802 is specifically used for:

[0164] Compare the summary information in the nodes of the first metadata tree and the second metadata tree;

[0165] Based on the comparison results, a first target leaf node is identified in the first metadata tree whose digest information is the same as that stored in the corresponding leaf node of the second metadata tree;

[0166] The data content of the leaf nodes other than the first target leaf node in the first metadata tree corresponding to the fragments in the file to be stored is determined as the difference data.

[0167] In one embodiment, the determining module 802, when comparing the summary information in the nodes of the first metadata tree and the second metadata tree, and determining the first target leaf node in the first metadata tree whose summary information is the same as that stored in the corresponding leaf node of the second metadata tree based on the comparison result, is specifically used for:

[0168] The summary information in the peer nodes of the first metadata tree and the second metadata tree is compared in a top-down manner; the number of subtree levels corresponding to the peer nodes is the same.

[0169] Based on the comparison results, a first target node is identified that stores the same digest information as the corresponding node in the second metadata tree in the first metadata tree;

[0170] Each leaf node contained in the first target node is determined as the first target leaf node.

[0171] In one embodiment, the leaf nodes in the file metadata tree are also used to store a first location index of the storage location of the fragment corresponding to the leaf node;

[0172] Storage module 803 is also used for:

[0173] After storing the difference data, a first location index of the storage location of the difference data is obtained, and the obtained first location index is stored in the leaf node corresponding to the difference data in the first metadata tree;

[0174] Obtain the first position index of the first target leaf node in the second target leaf node corresponding to the first target leaf node in the second metadata tree, and store the obtained first position index in the first target leaf node; the second target leaf node is a leaf node in the second metadata tree with the same summary information as the first target leaf node.

[0175] In one embodiment, the above-described apparatus further includes an access processing module, configured to:

[0176] In response to a file access request, the system obtains the first position index in each leaf node of the target metadata tree, and obtains the target fragments at each position indicated by each first position index; the target metadata tree is the metadata tree indicated by the file access request.

[0177] Merge each of the target fragments to obtain the corresponding merged result file;

[0178] The merged result file is sent back to the requesting end.

[0179] In one embodiment, the leaf nodes in the file metadata tree are also used to store a second position index of the content position of the fragment corresponding to the leaf node in its respective file;

[0180] When merging the target fragments, the access processing module specifically performs the following steps: obtaining the second position index corresponding to each target fragment from each leaf node of the target metadata tree; and merging the target fragments according to their respective second position indices.

[0181] In one embodiment, the determining module 802 is further configured to: obtain the second position index corresponding to each of the target fragments from each leaf node of the target metadata tree before obtaining the second data structure of the stored target file; and merge the target fragments according to the corresponding second position indexes.

[0182] The data processing apparatus disclosed in this application is relatively simple to describe because it corresponds to the data processing method disclosed in the above method embodiments. For any similarities, please refer to the description of the above method embodiments, which will not be detailed here.

[0183] This application also discloses an electronic device, the composition and structure of which are as follows: Figure 9 As shown, it includes at least:

[0184] Memory 10 is used to store the computer instruction set;

[0185] Computer instruction sets can be implemented in the form of computer programs.

[0186] The processor 20 is configured to implement the data processing method disclosed in any of the above method embodiments by executing a computer instruction set.

[0187] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU), or other programmable logic devices.

[0188] Electronic devices have a display device and / or have a display interface and can connect to an external display device.

[0189] Optionally, the electronic device may also include a camera assembly, and / or be connected to an external camera assembly.

[0190] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0191] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0192] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0193] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0194] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0195] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0196] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, comprising: Obtain the first metadata tree of the file to be stored, and obtain the second metadata tree of the stored target file; wherein, the file's metadata tree is a tree-shaped data structure for storing file fragment information, constructed by dividing the file into at least one fragment according to a set fragment size. A leaf node in the metadata tree corresponds to a fragment of the file, and the leaf node is used to store at least the summary information of the corresponding fragment. The non-leaf nodes are used to store the summary information of the merged result of the summary information in the corresponding child nodes; the summary information of the fragment is the hash value obtained by hashing the data content of the fragment using a preset hash algorithm; The digest information in the peer-level nodes of the first metadata tree and the second metadata tree is compared in a top-down manner. The peer-level node is the layer corresponding to the node whose subtrees contain the same number of layers between the two metadata trees, and the subtrees corresponding to the peer-level nodes all have the same number of layers. Based on the comparison results, a first target node is determined that stores the same digest information as the corresponding node in the second metadata tree in the first metadata tree. Each leaf node contained in the first target node is determined as the first target leaf node. The data content of the leaf nodes in the first metadata tree other than the first target leaf node, corresponding to the fragments in the file to be stored, is determined as the difference data. The difference data is the data in the file to be stored that is different from the target file. The difference data is stored.

2. The method according to claim 1, obtaining the first metadata tree of the file to be stored, includes: Construct the first metadata tree of the file to be stored; Obtaining the second metadata tree of the target file includes at least one of the following methods: Construct the second metadata tree of the target file; Read the second metadata tree of the target file from the preset storage.

3. The method according to claim 1, wherein the leaf nodes in the file metadata tree are further used to store a first location index of the storage location of the fragment corresponding to the leaf node; after storing the difference data, the method further includes: Obtain the first location index of the storage location of the difference data, and store the obtained first location index in the leaf node corresponding to the difference data in the first metadata tree; Obtain the first position index of the first target leaf node in the second target leaf node in the second metadata tree, and store the obtained first position index in the first target leaf node; The second target leaf node is a leaf node in the second metadata tree that has the same summary information as the first target leaf node.

4. The method according to claim 3, further comprising: In response to a file access request, the system obtains the first position index in each leaf node of the target metadata tree, and obtains the target fragments at each position indicated by each first position index; the target metadata tree is the metadata tree indicated by the file access request. Merge each of the target fragments to obtain the corresponding merged result file; The merged result file is sent back to the requesting end.

5. The method according to claim 4, wherein the leaf nodes in the file metadata tree are further used to store a second position index of the content position of the fragment corresponding to the leaf node in its respective file; the merging of the target fragments includes: Obtain the second position index corresponding to each target fragment from each leaf node of the target metadata tree; The target fragments are merged according to their respective second position indices.

6. The method according to any one of claims 1 to 5, further comprising, before obtaining the second metadata tree of the stored target file: Determine a first file from the existing stored files that has the same structural layout as the file to be stored; The target file is determined from the first file that meets similar conditions to the content of the file to be stored.

7. A data processing apparatus, comprising: The acquisition module is used to obtain the first metadata tree of the file to be stored and the second metadata tree of the stored target file. The file's metadata tree is a tree-like data structure constructed by dividing the file into at least one fragment according to a set fragment size. Each leaf node in the metadata tree corresponds to a fragment of the file. The leaf node stores at least the summary information of its corresponding fragment, while non-leaf nodes store the summary information of the merged result of the summary information from each corresponding child node. The fragment's summary information is a hash value obtained by hashing the data content of the fragment using a preset hash algorithm. The determination module is used to compare the summary information in the peer-level nodes of the first metadata tree and the second metadata tree in a top-down manner; the peer level is the layer corresponding to the nodes whose subtrees contain the same number of levels between the two metadata trees, and the subtrees corresponding to the peer-level nodes all have the same number of levels; based on the comparison results, a first target node in the first metadata tree that stores the same summary information as the corresponding node in the second metadata tree is determined; each leaf node contained in the first target node is determined as the first target leaf node; the data content of the leaf nodes in the first metadata tree other than the first target leaf node, corresponding to the fragments in the file to be stored, is determined as the difference data; the difference data is the data in the file to be stored that is different from the target file; A storage module is used to store the difference data.

Images