A data storage method and device, electronic equipment and readable storage medium
By pre-compiling the dependency indexes of data nodes and processing them in parallel, the problem of high computational resource consumption in the replication proof algorithm is solved, thereby improving storage efficiency and shortening the running time.
Patent Information
- Application Number
- CN202111091951.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-17
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2041-09-17
AI Technical Summary
Existing proof-of-replication algorithms consume a great deal of computational resources during the computation process, resulting in low efficiency.
By pre-compiling the extended dependency node indexes of all data nodes in the current storage layer and synchronously computing the dependency node indexes of this layer, and caching these indexes in the cache space, the dependent data of data nodes can be computed in parallel.
It improves the efficiency of data storage, shortens the running time, and saves computing resources.
Smart Images
Figure CN113946545B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data storage method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] Replication proof is a novel storage proof strategy. It allows a prover to provide a storage proof to a user, demonstrating that the user's data has been copied and stored on the prover's unique dedicated physical storage device. When a prover promises to store n distinct copies (independent physical copies) of certain data, they then provide proof to a verifier through a replication proof interaction protocol, proving that the prover has indeed stored each copy of the data. Mathematically, this algorithm is defined as a polynomial-time algorithm, assuming that a copy is difficult to generate quickly. The algorithm implementation process can include: 1. Setup phase: In this phase, the system sets common parameters; 2. Replication phase: In this phase, any one or more parties calculate and generate a unique sequence number based on the original data D; 3. Proof phase: The proof builder receives a challenge and must generate a proof that they possess a copy derived from the original data D. The proof builder can successfully meet the challenge if and only if they already possess this copy; that is, it is almost impossible for the proof builder to immediately generate a copy when receiving the challenge; 4. Verification phase: After receiving the proof, the verifier verifies the validity of the proof and decides whether to accept or reject it.
[0003] Currently, the commonly used replication proof algorithm is the stacked depth robust graph consensus mechanism algorithm. This algorithm divides the entire data sector (32 Gbit in this example) into data nodes in 32-bit units. The data processing of the sector is divided into 11 layers (lε[1,11]), calculated from top to bottom, with the upper layer becoming the expansion of the lower layer. The computation of a node depends on itself and some of its expansion nodes, where it depends on 6 nodes and its expansion nodes depend on 8 nodes. The nodes that a node depends on are the 6 nodes before the current node in the current computation node's layer, which must include the data to its left and nearest neighbor. The nodes that the expansion depends on are the 8 data nodes randomly selected from the layer above the current computation node's layer. The dependency relationship between a node and other nodes in a certain layer is as follows: Figure 2 The algorithm obtains the current node data by performing SHA256 calculation on the node data it depends on, the node data it extends to depend on, as well as the ID information and length information.
[0004] Because the data of a node at each level depends on the data of the six nodes preceding it, especially its nearest left node, calculations at the same level must be performed sequentially from left to right. Since lower-level nodes depend on the data of higher-level nodes, calculations across different levels must be performed sequentially from top to bottom. This process involves calculating dependencies and reading data from dependent nodes, making it extremely computationally intensive. Summary of the Invention
[0005] This application provides a data storage method, apparatus, electronic device, and readable storage medium to solve the problem that the replication proof algorithm in the prior art consumes extremely high computational resources.
[0006] In a first aspect, embodiments of this application provide a data storage method, the method comprising the following steps:
[0007] During the storage process of the data to be stored, the index of the extended dependent node corresponding to the extended dependent node of all data nodes in the current storage layer is pre-calculated, and the index of the local dependent node corresponding to the local dependent node of multiple nodes is calculated synchronously.
[0008] The extended dependency node index is cached in the internal extended dependency node cache space;
[0009] Based on the cached extended dependency node index, extract the extended dependency node data corresponding to the extended dependency node from the external storage space corresponding to the extended layer, and synchronously obtain the local dependency node data corresponding to the local dependency node based on the local dependency node index.
[0010] The extended dependency node data and the local dependency node data are cached in the memory cache space.
[0011] Optionally, the step of pre-calculating the extended dependency node indices corresponding to the extended dependency nodes of all data nodes in the current storage layer includes:
[0012] Based on the Feistel encryption algorithm, the indexes of the extended dependent nodes corresponding to the extended dependent nodes of all data nodes in the current storage layer are calculated.
[0013] Optionally, the synchronous calculation of the indexes of the dependent nodes corresponding to the dependent nodes of the current layer for multiple nodes includes:
[0014] The chacha8 algorithm is used to synchronously calculate the indices of the dependent nodes of the current layer for multiple nodes.
[0015] Optionally, caching the extended dependency node data and the local dependency node data in the memory cache space includes:
[0016] During the process of caching the extended dependency node data and the local dependency node data, it is determined whether the amount of cached data has reached the data amount threshold.
[0017] If so, reduce the data caching speed and stop caching when the cache space is full.
[0018] Secondly, embodiments of this application provide a data storage device, the device comprising the following modules:
[0019] The dependency node index calculation module is used to pre-calculate the extended dependency node index corresponding to the extended dependency node of all data nodes in the current storage layer during the storage process of the data to be stored, and synchronously calculate the local dependency node index corresponding to the local dependency node of multiple nodes.
[0020] An extended dependency node index caching module is used to cache the extended dependency node index in the internal extended dependency node cache space;
[0021] The dependency node data acquisition module is used to extract the extended dependency node data corresponding to the extended dependency node from the external storage space corresponding to the extended layer according to the cached extended dependency node index, and synchronously acquire the local dependency node data corresponding to the local dependency node according to the local dependency node index.
[0022] The dependency node data caching module is used to cache the extended dependency node data and the current layer dependency node data in the memory cache space.
[0023] Optionally, the dependency node index calculation module includes:
[0024] The extended dependency node index calculation unit is used to calculate the extended dependency node index corresponding to the extended dependency node of all data nodes in the current storage layer based on the Feistel encryption algorithm.
[0025] Optionally, the dependency node index calculation module includes:
[0026] The local layer dependency node index calculation unit is used to synchronously calculate the local layer dependency node index of multiple nodes based on the chacha8 algorithm.
[0027] Optionally, the dependent node data caching module includes:
[0028] The cached data volume determination unit is used to determine whether the cached data volume reaches the data volume threshold during the process of caching the extended dependency node data and the local dependency node data.
[0029] The cached data reduction unit is used to reduce the data caching speed when the amount of cached data reaches the data volume threshold, and to stop caching when the cache space is full.
[0030] Thirdly, embodiments of this application provide an electronic device, including:
[0031] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of any of the above-described data storage methods.
[0032] Fourthly, embodiments of this application provide a readable storage medium that, when the instructions in the readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the steps of any of the data storage methods described above.
[0033] Compared with the prior art, this application has the following advantages:
[0034] This application provides a data storage method, apparatus, electronic device, and readable storage medium. During the storage process of data to be stored, the extended dependency node indices corresponding to the extended dependency nodes of all data nodes in the current storage layer are pre-calculated, and the local dependency node indices corresponding to the local dependency nodes of multiple nodes are simultaneously calculated. These extended dependency node indices are cached in an internal extended dependency node cache space. Based on the cached extended dependency node indices, the extended dependency node data corresponding to the extended dependency nodes is extracted from the external storage space corresponding to the extended layer. Simultaneously, based on the local dependency node index, the local dependency node data corresponding to the local dependency node is obtained. Both the extended dependency node data and the local dependency node data are cached in a memory cache space. By pre-calculating the extended dependency nodes of all nodes, this application allows for parallel computation of multiple computational steps during the replication phase, changing them from sequential execution to parallel computation. This improves operational efficiency, shortens runtime, and saves computational resources. Attached Figure Description
[0035] Figure 1 A flowchart illustrating the steps of a data storage method provided in this application embodiment;
[0036] Figure 2 A schematic diagram of a self-dependent node provided in an embodiment of this application;
[0037] Figure 3 A schematic diagram of a chip architecture provided in an embodiment of this application;
[0038] Figure 4 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this application. Detailed Implementation
[0039] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0040] Example 1
[0041] Reference Figure 1 The flowchart illustrates the steps of a data storage method provided in an embodiment of this application, as shown below. Figure 1 As shown, this data storage method may specifically include the following steps:
[0042] Step 101: During the storage process of the data to be stored, pre-calculate the extended dependency node index corresponding to the extended dependency node of all data nodes in the current storage layer, and simultaneously calculate the local dependency node index corresponding to the local dependency node of multiple nodes.
[0043] The embodiments of this application can be applied to scenarios where multiple computational steps in the replication stage are changed from sequential execution to parallel computation.
[0044] The data to be stored refers to the data intended for storage. The data type of the data to be stored can be determined according to business needs, and this embodiment does not impose any restrictions on it.
[0045] During the storage process of the data to be stored, the extended dependency node indexes corresponding to the extended dependency nodes of all data nodes in the current storage layer can be pre-calculated, and the local dependency node indexes corresponding to the local dependency nodes of multiple nodes can be calculated simultaneously.
[0046] In this example, during the replication process, the calculation of the extended dependency node index uses the Feistel encryption algorithm, and the parameters are related to the node index and sequence number, and do not depend on other dynamic parameters.
[0047] The computation of its own dependent nodes depends on the 6 nodes preceding the current node, and is derived using the chacha8 encryption algorithm. Its parameter node index is related to a specific seed and does not depend on other dynamic parameters.
[0048] After synchronously calculating the indexes of the current layer's dependent nodes and the extended dependent nodes, proceed to step 102.
[0049] Step 102: Cache the extended dependency node index in the internal extended dependency node cache space.
[0050] After synchronously calculating the indexes of the current layer's dependent nodes and the extended dependent nodes, the extended dependent node indexes can be cached in the internal extended dependent node cache space.
[0051] After caching the extended dependency node index in the internal extended dependency node cache space, proceed to step 103.
[0052] Step 103: Based on the cached extended dependency node index, extract the extended dependency node data corresponding to the extended dependency node from the external storage space corresponding to the extended layer, and synchronously obtain the local dependency node data corresponding to the local dependency node based on the local dependency node index.
[0053] After synchronously calculating the indices of the current layer's dependent nodes and the extended dependent nodes, the extended dependent node data corresponding to the extended dependent nodes can be extracted from the external storage space corresponding to the extended layer based on the cached extended dependent node index. Simultaneously, the current layer's dependent node data corresponding to the current layer's dependent nodes can be obtained based on the current layer's dependent node index.
[0054] After synchronously obtaining the extended dependency node data and the current layer dependency node data, proceed to step 104.
[0055] Step 104: Cache the extended dependency node data and the local dependency node data in the memory cache space.
[0056] After synchronously acquiring the extended dependency node data and the current layer dependency node data, the extended dependency node data and the current layer dependency node data can be cached in the internal cache space. During the caching process of dependency node data, if the amount of cached data reaches the data amount threshold, the data caching speed can be reduced, and caching can be stopped when the cache control is full. Specifically, when the dependency node cache is about to overflow, the data filling speed is slowed down, and filling is stopped when the cache is full.
[0057] The above process can be described by referring to the following steps:
[0058] 1. Pre-calculate the extended dependency nodes of all nodes and place the calculated nodes in the extended dependency node index cache for later use;
[0059] 1.a. Based on the extended dependency node index of the cache, extract the corresponding data node data from the external storage and populate the dependency node data cache. This step can be performed synchronously with step 1 above.
[0060] 1.b. Calculate the self-dependent nodes of multiple nodes and obtain the self-dependent node indexes;
[0061] 1.c. Extract the corresponding dependent node data from external storage and fill it into the dependent node data cache; this data is stored together with the cached data in 1.a.
[0062] Steps 1.a, 1.b, and 1.c above can be performed simultaneously.
[0063] When extracting data from its dependent nodes in step 1.c, some of the nearest nodes may not have finished computing yet, so the corresponding positions in the cache can be marked as empty; once the corresponding nodes have finished computing, these positions can be filled; for example... Figure 2 As shown, the data of the dependent nodes of node a can be pre-read. When the data of the dependent nodes of node b is pre-read, the calculation of node a has not yet been completed. When the data of the dependent nodes of node c is pre-read, the calculations of nodes a and b have not yet been completed, so they cannot be pre-read.
[0064] The synchronization in steps 1.a-1.c is handled by the synchronization controller. When the cache of a dependent node is about to overflow, the data filling speed is slowed down, and filling is stopped when the cache is full.
[0065] The solution in this application embodiment can be implemented using a chip architecture, which can be combined with... Figure 3 The following is a detailed description.
[0066] like Figure 3 The image shows the chip architecture.
[0067] 10 is the extended dependency node calculation module, which may include the feistl encryption module, responsible for calculating all extended dependency nodes and storing the results in cache 70.
[0068] 20 is the self-dependent node calculation module. This module may include the chacha8 encryption module, which is responsible for calculating its own dependent nodes and storing the results in cache 80.
[0069] 30 is external storage, which stores copies of the data that need to be stored and intermediate data during the data copying process; after obtaining the dependent node index, 30 pre-reads the data and outputs it to 40.
[0070] 40 is a cache for dependent node data. The data structure of this module can be, but is not limited to, a FIFO queue or a ring. Each node on it stores the following information:
[0071] 1. ID information, including replica ID and node number;
[0072] 2. Node data it depends on;
[0073] 3. Expand the dependent node data;
[0074] 4. Length information.
[0075] If the data to be pre-read is not yet ready, mark the location of the data as empty. After all the dependent node data corresponding to a node enters the cache, subsequent SHA256 operations can be performed.
[0076] 50 is the SHA256 calculation module. This module can extract pre-read node data from 40, perform SHA256 calculation, output the calculation result to 30, and simultaneously fill in the corresponding previously empty positions in 40.
[0077] 60 is a synchronous processing controller. This module can monitor the space usage of the 40 buffer and the calculation speed of 50, adjust the calculation speed of 10 and 20, adjust the speed of filling the 40 and 70 buffers, and ensure that all the required data has been pre-read to 40 before 50 calculates a certain node.
[0078] 70 is an extended dependency node cache, which can be responsible for caching the indexes of extended dependency data nodes in the data above each node. This data is used throughout the replication phase.
[0079] 80 is a cache for its own dependent nodes, which can be responsible for caching the indexes of the data nodes that each node depends on in the data of this layer.
[0080] The data storage method provided in this application pre-calculates the extended dependency node indices corresponding to the extended dependency nodes of all data nodes in the current storage layer during the storage process of the data to be stored, and simultaneously calculates the local dependency node indices corresponding to the local dependency nodes of multiple nodes. The extended dependency node indices are cached in an internal extended dependency node cache space. Based on the cached extended dependency node indices, the extended dependency node data corresponding to the extended dependency nodes is extracted from the external storage space corresponding to the extended layer. Simultaneously, based on the local dependency node index, the local dependency node data corresponding to the local dependency node is obtained. Both the extended dependency node data and the local dependency node data are cached in a memory cache space. By pre-calculating the extended dependency nodes of all nodes, this application embodiment can change the sequential execution of multiple computational steps within the replication phase to parallel computation during data storage, thereby improving operating efficiency, shortening running time, and saving computing resources.
[0081] Example 2
[0082] Reference Figure 4 The diagram shows a structural schematic of a data storage device provided in an embodiment of this application, such as... Figure 4 As shown, the data storage device 400 may specifically include the following modules:
[0083] The dependency node index calculation module 410 is used to pre-calculate the extended dependency node index corresponding to the extended dependency node of all data nodes in the current storage layer during the storage process of the data to be stored, and synchronously calculate the local dependency node index corresponding to the local dependency node of multiple nodes.
[0084] The extended dependency node index caching module 420 is used to cache the extended dependency node index in the internal extended dependency node cache space;
[0085] The dependency node data acquisition module 430 is used to extract the extended dependency node data corresponding to the extended dependency node from the external storage space corresponding to the extended layer according to the cached extended dependency node index, and synchronously acquire the local dependency node data corresponding to the local dependency node according to the local dependency node index.
[0086] The dependency node data caching module 440 is used to cache the extended dependency node data and the current layer dependency node data in the memory cache space.
[0087] Optionally, the dependency node index calculation module 410 includes:
[0088] The extended dependency node index calculation unit is used to calculate the extended dependency node index corresponding to the extended dependency node of all data nodes in the current storage layer based on the Feistel encryption algorithm.
[0089] Optionally, the dependency node index calculation module 420 includes:
[0090] The local layer dependency node index calculation unit is used to synchronously calculate the local layer dependency node index of multiple nodes based on the chacha8 algorithm.
[0091] Optionally, the dependent node data caching module includes:
[0092] The cached data volume determination unit is used to determine whether the cached data volume reaches the data volume threshold during the process of caching the extended dependency node data and the local dependency node data.
[0093] The cached data reduction unit is used to reduce the data caching speed when the amount of cached data reaches the data volume threshold, and to stop caching when the cache space is full.
[0094] The data storage device provided in this application embodiment pre-calculates the extended dependency node indices corresponding to the extended dependency nodes of all data nodes in the current storage layer during the storage process of the data to be stored, and simultaneously calculates the local dependency node indices corresponding to the local dependency nodes of multiple nodes. The extended dependency node indices are cached in an internal extended dependency node cache space. Based on the cached extended dependency node indices, the extended dependency node data corresponding to the extended dependency nodes is extracted from the external storage space corresponding to the extended layer, and the local dependency node data corresponding to the local dependency node is synchronously obtained based on the local dependency node index. Both the extended dependency node data and the local dependency node data are cached in a memory cache space. By pre-calculating the extended dependency nodes of all nodes, this application embodiment can change the sequential execution of multiple computational steps within the replication phase to parallel computation during data storage, thereby improving operating efficiency, shortening running time, and saving computing resources.
[0095] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0096] Additionally, embodiments of this application also provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the data storage methods described above.
[0097] This application also provides a readable storage medium, which, when the instructions in the readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the steps of any of the above-described data storage methods.
[0098] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0099] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0100] The present application provides a detailed description of a data storage method and a data storage device. Specific examples have been used to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present application. At the same time, those skilled in the art will know that there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.
Claims
1. A data storage method, characterized in that, The method includes the following steps: During the storage process of the data to be stored, the index of the extended dependent node corresponding to the extended dependent node of all data nodes in the current storage layer is pre-calculated, and the index of the local dependent node corresponding to the local dependent node of multiple nodes is calculated synchronously. The extended dependency node index is cached in the internal extended dependency node cache space; Based on the cached extended dependency node index, extract the extended dependency node data corresponding to the extended dependency node from the external storage space corresponding to the extended layer, and synchronously obtain the local dependency node data corresponding to the local dependency node based on the local dependency node index. The extended dependency node data and the local dependency node data are cached in the memory cache space; The pre-calculation of the extended dependency node indices corresponding to the extended dependency nodes of all data nodes in the current storage layer includes: Based on the Feistel encryption algorithm, the indexes of the extended dependent nodes corresponding to the extended dependent nodes of all data nodes in the current storage layer are calculated.
2. The method according to claim 1, characterized in that, The synchronous calculation obtains the indexes of the dependent nodes at the current layer corresponding to the dependent nodes of multiple nodes, including: The chacha8 algorithm is used to synchronously calculate the indices of the dependent nodes of the current layer for multiple nodes.
3. The method according to claim 1, characterized in that, The step of caching the extended dependency node data and the local dependency node data in the memory cache space includes: During the process of caching the extended dependency node data and the local dependency node data, it is determined whether the amount of cached data has reached the data amount threshold. If so, reduce the data caching speed and stop caching when the cache space is full.
4. A data storage device, characterized in that, The device includes the following modules: The dependency node index calculation module is used to pre-calculate the extended dependency node index corresponding to the extended dependency node of all data nodes in the current storage layer during the storage process of the data to be stored, and synchronously calculate the local dependency node index corresponding to the local dependency node of multiple nodes. An extended dependency node index caching module is used to cache the extended dependency node index in the internal extended dependency node cache space; The dependency node data acquisition module is used to extract the extended dependency node data corresponding to the extended dependency node from the external storage space corresponding to the extended layer according to the cached extended dependency node index, and synchronously acquire the local dependency node data corresponding to the local dependency node according to the local dependency node index. A dependency node data caching module is used to cache the extended dependency node data and the current layer dependency node data in the memory cache space; The dependency node index calculation module includes: The extended dependency node index calculation unit is used to calculate the extended dependency node index corresponding to the extended dependency node of all data nodes in the current storage layer based on the Feistel encryption algorithm.
5. The apparatus according to claim 4, characterized in that, The dependency node index calculation module includes: The local layer dependency node index calculation unit is used to synchronously calculate the local layer dependency node index of multiple nodes based on the chacha8 algorithm.
6. The apparatus according to claim 4, characterized in that, The dependent node data caching module includes: The cached data volume determination unit is used to determine whether the cached data volume reaches the data volume threshold during the process of caching the extended dependency node data and the local dependency node data. The cached data reduction unit is used to reduce the data caching speed when the amount of cached data reaches the data volume threshold, and to stop caching when the cache space is full.
7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data storage method as described in any one of claims 1 to 3.
8. A readable storage medium, characterized in that, When the instructions in the readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the steps of the data storage method according to any one of claims 1 to 3.
Citation Information
Patent Citations
Visualization method, device and system for dependence relation of global task nodes
CN106293928A