Data processing method and device, electronic equipment, storage medium and program product

Abstract

The present application provides a kind of data processing method, device, electronic equipment, storage medium and program product, the data processing method includes: plaintext data content is divided into multiple data blocks, and the block pointer of each data block is determined, the block pointer of data block includes: the encryption key of data block and the hash value of encrypted data block, encryption key is the hash value of data block, and encrypted data block is obtained by using encryption key to encrypt data block;Each encrypted data block and corresponding block pointer are stored;Create search tree, each node of search tree corresponds an encrypted data block, the information of node includes: the name of encrypted data block, the pointer for pointing to the storage location of encrypted data block, the pointer for indicating the child node of node;The name of encrypted data block is the hash value of data block;The load metric information of each node of search tree is obtained, and whether the structure of search tree is adjusted according to load metric information.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device, storage medium and program product. Background Technology

[0002] With society placing increasing emphasis on personal data privacy, the need for secure data storage and distribution is particularly urgent. Current technologies typically encrypt and store the entire file, making key generation, distribution, storage, and updating complex and vulnerable to attack. Furthermore, sharing the entire encrypted file for privacy data distribution lacks flexibility. Additionally, the efficiency of data retrieval from encrypted files needs further improvement. Summary of the Invention

[0003] This invention provides a data processing method, apparatus, electronic device, storage medium, and program product to address the problems of existing privacy data storage schemes, such as complex key management, susceptibility to attacks, insufficient flexibility during data flow, and low data query efficiency.

[0004] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0005] In a first aspect, embodiments of the present invention provide a data processing method, including:

[0006] The plaintext data content is divided into multiple data blocks, and a block pointer for each data block is determined. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key.

[0007] Store each of the encrypted data blocks and the corresponding block pointers;

[0008] Create a search tree, where each node of the search tree corresponds to one of the encrypted data blocks. The information of the node includes: the name of the encrypted data block, a pointer to the storage location of the encrypted data block, and pointers to the child nodes of the node; the name of the encrypted data block is the hash value of the data block.

[0009] Obtain the load metric information of each node in the search tree, and determine whether to adjust the structure of the search tree based on the load metric information.

[0010] Optionally, the load metric information includes: the load of the node at the next time point;

[0011] The process of obtaining load metric information for each node of the search tree includes:

[0012] The load of the node at the current time is determined based on the access frequency of the node at the current time and the first weight coefficient corresponding to the access frequency, the number of child nodes of the node at the current time and the second weight coefficient corresponding to the number of child nodes.

[0013] Based on the node's load at the current time and its historical load, predict the node's load at the next time.

[0014] The process of determining whether to adjust the structure of the search tree based on the load metric information includes:

[0015] If the load of the node at the next time point is greater than or equal to a first threshold, it is determined to adjust the structure of the search tree, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, reconstructing the search tree, or splitting the node.

[0016] Optionally, the load metric information includes: the load difference between the node and its neighboring nodes;

[0017] The process of obtaining load metric information for each node of the search tree includes:

[0018] The load of a node is determined based on the node’s access frequency and the first weight coefficient corresponding to the access frequency, the number of the node’s child nodes and the second weight coefficient corresponding to the number of child nodes.

[0019] The load of the adjacent nodes is determined based on the access frequency of the adjacent nodes and the first weight coefficient corresponding to the access frequency, as well as the number of child nodes of the adjacent nodes and the second weight coefficient corresponding to the number of child nodes.

[0020] Calculate the load difference between the node and its neighboring nodes based on the node's load and the load of the neighboring nodes;

[0021] The process of determining whether to adjust the structure of the search tree based on the load metric information includes:

[0022] If the load difference between the node and its neighboring nodes is greater than or equal to a second threshold, the structure of the search tree is determined to be adjusted, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, or reconstructing the search tree.

[0023] Optional, also includes:

[0024] The first weight coefficient and the second weight coefficient are updated according to the preset adjustment rate and performance evaluation function. The performance evaluation function is determined based on the efficiency index and the variability index. The efficiency index is determined based on the average response time of all nodes in the search tree, and the variability index is determined based on the response time of each node in the search tree and the average response time.

[0025] Optionally, the load metric information includes: the number of child nodes of the node;

[0026] The process of determining whether to adjust the structure of the search tree based on the load metric information includes:

[0027] When the number of child nodes of a node exceeds a preset maximum number of child nodes, the structure of the search tree is adjusted, wherein adjusting the structure of the search tree includes splitting the node;

[0028] When the number of child nodes of a node is less than a preset minimum number of child nodes, the structure of the search tree is adjusted, wherein adjusting the structure of the search tree includes merging the node with its neighboring nodes.

[0029] Optional, also includes:

[0030] Multiple data objects are defined for the plaintext data content, and the types of the data objects include root directory, directory, subdirectory, file and / or data;

[0031] An initial version object is created for each data object, and the version object is associated with multiple data blocks and multiple corresponding encrypted data blocks corresponding to the data object;

[0032] If the data object is modified, a new version object is created for the data object.

[0033] Each initial version object can be created separately. Optionally, the following can also be included:

[0034] A directed acyclic graph is constructed based on the version objects corresponding to the multiple data objects, wherein each node of the directed acyclic graph represents a version object of the data object, the edges of the directed acyclic graph represent the parent-child relationship between the data objects, and the information of the node includes the version ID;

[0035] When the data object is modified, a new node is created in the directed acyclic graph. The new node corresponds to the new version object of the data object and serves as a child node of the node of the initial version object of the data object. The information of the new node includes a version ID and a parent version ID, where the parent version ID is the version ID of the node of the initial version object of the data object.

[0036] Optional, also includes:

[0037] The encrypted data block is shared between the data provider and the data manager.

[0038] Secondly, embodiments of the present invention provide a data processing method, including:

[0039] Encrypted data blocks provided by the data provider are provided to the data consumer through a data circulation network. The encrypted data blocks are obtained and stored by the data provider through the following method: dividing the plaintext data content into multiple data blocks and determining the block pointer of each data block. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block. The encrypted data block is obtained by encrypting the data block using the encryption key.

[0040] Optional, also includes:

[0041] Construct a supply function that describes the amount of data the data manager is willing to provide for the benefit gained from each unit of data circulation;

[0042] The supply function is provided to the data circulation network, which then determines the optimal benefit that the data manager can obtain from the data circulation network for each unit of data circulation based on the supply function.

[0043] Thirdly, embodiments of the present invention provide a data processing apparatus, including:

[0044] An encryption module is used to divide plaintext data content into multiple data blocks and determine a block pointer for each data block. The block pointer of each data block includes: an encryption key for the data block and a hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key.

[0045] The storage module is used to store each of the encrypted data blocks and the corresponding block pointers;

[0046] The search tree creation module is used to create a search tree, where each node of the search tree corresponds to one of the encrypted data blocks. The information of the node includes: the name of the encrypted data block, a pointer to the storage location of the encrypted data block, and pointers to the child nodes of the node; the name of the encrypted data block is the hash value of the data block.

[0047] The search tree adjustment module is used to obtain the load metric information of each node of the search tree, and determine whether to adjust the structure of the search tree based on the load metric information.

[0048] Fourthly, embodiments of the present invention provide a data processing apparatus, comprising:

[0049] The management module is used to provide encrypted data blocks provided by the data provider to the data consumer through a data circulation network. The encrypted data blocks are obtained and stored by the data provider through the following method: dividing plaintext data content into multiple data blocks and determining a block pointer for each data block. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key.

[0050] Fifthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data processing method as described in the first or second aspect above.

[0051] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method as described in the first or second aspect above.

[0052] In a seventh aspect, embodiments of the present invention provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data processing method as described in the first or second aspect above.

[0053] In this embodiment of the invention, firstly, the plaintext data content is divided into multiple data blocks, and a block pointer for each data block is generated. The block pointer includes the hash value of the data block and the hash value of the encrypted data block. The hash value of the data block is used as the encryption key to encrypt the data block, ensuring that the data block is encrypted during transmission and storage. Since the hash value of the data block itself is used as the encryption key, there is no need for key generation, distribution, and updating management, reducing the possibility of data attacks. The hash value of the encrypted data block can also be used for integrity verification of the encrypted data block. Furthermore, during data circulation, only data blocks can be shared, achieving fine-grained data sharing and improving the flexibility of data circulation. Moreover, to facilitate rapid data block retrieval, a search tree is created. Each node of the search tree corresponds to an encrypted data block. To optimize the response time of the search tree, the structure of the search tree is adjusted based on the load metric information of each node, thereby improving the query efficiency of the search tree. Attached Figure Description

[0054] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0055] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the overall system architecture according to an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of a cached encrypted data block according to an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of a version control method under a directory structure according to an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of a version object according to an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the data storage structure of a directed acyclic graph according to an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the version control method according to an embodiment of the present invention;

[0062] Figure 8 This is a schematic flowchart of a data processing method according to another embodiment of the present invention;

[0063] Figure 9 This is a schematic diagram of the data flow model entity in an embodiment of the present invention;

[0064] Figure 10 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present invention;

[0065] Figure 11 This is a schematic diagram of the structure of a data processing device according to another embodiment of the present invention;

[0066] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Please refer to Figure 1 This invention provides a data processing method, including:

[0069] Step S11: Divide the plaintext data content into multiple data blocks and determine the block pointer of each data block. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key.

[0070] The plaintext data content may include: root directory, directory, subdirectory, file and / or data. The plaintext data content may be private data.

[0071] Dividing plaintext data content into multiple data blocks can be achieved by dividing the plaintext data content into multiple fixed-size data blocks. Optionally, the size of the data block can be a multiple of the common physical sector size, such as 4kB, and can be changed according to different storage settings. In some embodiments, the plaintext data content can be divided into data blocks of a fixed size, and if the size of the last data block does not reach the fixed size, information can be randomly padded to the fixed size. In some embodiments, each data block may also include the plaintext content divided into plaintext data content and randomly padded information.

[0072] In this embodiment of the invention, the pointer block can be represented as a 2-tuple. :

[0073]

[0074]

[0075] Where C represents the plaintext content of the data block, hash represents the cryptographic hash function, and Encrypt represents the symmetric encryption algorithm. The hash value of the encrypted data block... Its function is to perform data integrity verification on encrypted data blocks.

[0076] In this embodiment of the invention, the block pointer can be used to decrypt the content of the encrypted data block, but cannot modify the content of the data block, because the data block is fixed and immutable; changing any byte in the data block will change the encryption key of the data block. This also changes the version of the encrypted data block, thereby further altering its cryptographic hash value. .

[0077] In this embodiment of the invention, the block pointer information may include an encryption key. Metadata for hash functions and encryption algorithms.

[0078] In this embodiment of the invention, the plaintext content block (data block) appears temporarily and briefly in memory, and will not appear in other scenarios (storage and transmission, etc.), thereby improving the privacy and confidentiality of the data.

[0079] Step S12: Store each of the encrypted data blocks and the corresponding block pointers;

[0080] Please refer to Figure 2 and Figure 3 In this embodiment of the invention, a combination of near storage and far storage can be used to store encrypted data blocks in the cache. Near storage involves storing the cached encrypted data blocks in memory, which is faster than far storage. Far storage supports storage based on file, network, or cloud storage methods.

[0081] The process of writing encrypted data blocks is as follows: first, they are saved in near storage, then their logs are recorded to disk files, and the encrypted data blocks recorded in the logs are stored in far storage through background processing.

[0082] The process of reading encrypted data blocks is as follows: First, the encrypted data block is read from the near storage. If it does not exist (e.g., the encrypted data block is cleared from the near storage to save content space), then the encrypted data block is read from the far storage.

[0083] When storing encrypted data blocks using the above storage method, a cache inconsistency problem arises: the content of a data block may be updated in the cache but not elsewhere. In this embodiment of the invention, the cache inconsistency problem is simplified into a version control problem. The basic principle for solving the version control problem is that even the slightest inconsistency in content will result in encrypted data blocks having different names (in this embodiment, the hash value of the data block can be used as the name of the corresponding encrypted data block), thus ensuring complete consistency of an encrypted data block between two storage locations. The specific details of the version control scheme can be found in the descriptions of subsequent embodiments.

[0084] Step S13: Create a search tree, where each node of the search tree corresponds to one of the encrypted data blocks. The information of the node includes: the name of the encrypted data block, a pointer to the storage location of the encrypted data block, and pointers to the child nodes of the node; the name of the encrypted data block is the hash value of the data block.

[0085] In this embodiment of the invention, the search tree is used for retrieving encrypted data blocks.

[0086] In this embodiment of the invention, the nodes of the search tree are defined as follows:

[0087] The key value can be the name of the encrypted data block. In this embodiment of the invention, optionally, the name of the encrypted data block is the hash value of the data block.

[0088] The data includes: a pointer to the storage location of the encrypted data block, and related metadata information (such as the size of the encrypted data block, encryption algorithm, hash function, etc.).

[0089] Pointers used to indicate the child nodes of the node: references pointing to the left and right child nodes;

[0090] Metadata management: Stores metadata information, including the size of encrypted data blocks, encryption algorithms, hash functions, etc., for easy use during retrieval and decryption.

[0091] Step S14: Obtain the load metric information of each node of the search tree, and determine whether to adjust the structure of the search tree based on the load metric information.

[0092] To improve the insertion and query performance of the search tree, it is necessary to maintain the structural balance of the search tree. In this embodiment of the invention, the load metric information of each node of the search tree is monitored and / or predicted and analyzed in real time, and the structure of the search tree is dynamically adjusted to adapt to changes in data access.

[0093] In this embodiment of the invention, firstly, the plaintext data content is divided into multiple data blocks, and a block pointer for each data block is generated. The block pointer includes the hash value of the data block and the hash value of the encrypted data block. The hash value of the data block is used as the encryption key to encrypt the data block, ensuring that the data block is encrypted during transmission and storage. Since the hash value of the data block itself is used as the encryption key, there is no need for key generation, distribution, and updating management, reducing the possibility of data attacks. The hash value of the encrypted data block can also be used for integrity verification of the encrypted data block. Furthermore, during data circulation, only data blocks can be shared, achieving fine-grained data sharing and improving the flexibility of data circulation. Moreover, to facilitate rapid data block retrieval, a search tree is created. Each node of the search tree corresponds to an encrypted data block. To optimize the response time of the search tree, the structure of the search tree is adjusted based on the load metric information of each node, thereby improving the query efficiency of the search tree.

[0094] In some embodiments, optionally, the load metric information includes: the load of the node at the next time point; wherein, obtaining the load metric information of each node of the search tree includes:

[0095] Step S141a: Determine the load of the node at the current time point based on the access frequency of the node at the current time point and the first weight coefficient corresponding to the access frequency, as well as the number of child nodes of the node at the current time point and the second weight coefficient corresponding to the number of child nodes.

[0096] In this embodiment of the invention, optionally, the load of the node can be determined using the following formula:

[0097]

[0098] in, This represents the load of node n. This represents the number of child nodes of node n. This represents the frequency of visits to node n. This represents the second weighting coefficient. This represents the first weighting coefficient. and This setting adjusts the importance of the number of child nodes and access frequency in load balancing, allowing adjustments based on actual needs. For example, if response speed is a greater concern, the number of child nodes can be increased. The value of .

[0099] Step S142a: Based on the node's load at the current time point and the node's historical load, predict the node's load at the next time point;

[0100] Predicting future load is crucial for making proactive decisions about balancing the search tree. By forecasting, potential overloads of the search tree can be prevented, rather than simply reacting to events that have already occurred.

[0101] In this embodiment of the invention, the weighted moving average method can be used to predict the load of a node at the next time point:

[0102]

[0103] in, This represents the predicted load of node n at time t+1 (i.e., the next time point). This represents the actual load of node n at the current time t. This represents the historical load (e.g., historical average load) of node n. A smoothing factor is used to balance the effects of historical forecasts and current actual loads. Smoothing factor This setting can be based on the historical load volatility. If the historical load does not change significantly, It can be set higher to reduce forecast fluctuations; conversely, if historical loads change drastically, It should be set to a lower value.

[0104] The process of determining whether to adjust the structure of the search tree based on the load metric information includes:

[0105] Step S142a: If the load of the node at the next time point is greater than or equal to the first threshold, determine to adjust the structure of the search tree, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, reconstructing the search tree, and splitting the node.

[0106] In some embodiments, optionally, the load metric information includes: the load difference between the node and its neighboring nodes; wherein, obtaining the load metric information of each node in the search tree includes:

[0107] Step S141b: Determine the load of the node based on the node's access frequency and the first weight coefficient corresponding to the access frequency, the number of the node's child nodes and the second weight coefficient corresponding to the number of child nodes;

[0108] Similarly, the load of the node can be determined using the following formula: .

[0109] Step S142b: Determine the load of the adjacent node based on the access frequency of the adjacent node and the first weight coefficient corresponding to the access frequency, the number of child nodes of the adjacent node and the second weight coefficient corresponding to the number of child nodes;

[0110] In this embodiment of the invention, a node's neighboring node refers to a node that shares the same parent node as the node in question, and the number of neighboring nodes can be one or more. If there are multiple neighboring nodes, the load of each of the multiple neighboring nodes is determined.

[0111] Step S143b: Calculate the load difference between the node and its neighboring nodes based on the load of the node and the load of the neighboring nodes;

[0112] By comparing the load differences between adjacent nodes, it is possible to identify which nodes require balancing operations. In this embodiment of the invention, not all load imbalances require immediate attention; measures are only taken when the load difference exceeds a certain threshold.

[0113] In this embodiment of the invention, the load difference between a node and its neighboring nodes can be determined using the following formula:

[0114]

[0115] in, This represents the load difference of node n. This represents the actual load of node n. This represents the average load of a node's neighboring nodes. The load difference is used to determine whether adjustments to the search tree structure are needed.

[0116] The process of determining whether to adjust the structure of the search tree based on the load metric information includes:

[0117] Step S144b: If the load difference between the node and its neighboring nodes is greater than or equal to a second threshold, determine to adjust the structure of the search tree, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, or reconstructing the search tree. The rotation operation can be determined based on the direction and magnitude of the load difference. For example, if the right subtree of a node is too heavy, a left rotation may be necessary to balance it.

[0118] In this embodiment of the invention, the operation type can be adaptively selected based on load differences, i.e., deciding whether to perform a local tree rotation or reconstruct the search tree. Adaptively selecting the operation type can reduce unnecessary computation and improve algorithm efficiency.

[0119] In this embodiment of the invention, the second threshold can be determined using the following formula:

[0120]

[0121] Where T represents the second threshold of load difference. The threshold coefficient is determined based on historical performance data (such as response time). This represents the average load on the search tree. The value is adjusted based on performance targets and historical performance data.

[0122] In some embodiments, optionally, the method further includes: updating the first weight coefficient and the second weight coefficient according to a preset adjustment rate and performance evaluation function, wherein the performance evaluation function is determined based on an efficiency index and a variability index, the efficiency index is determined based on the average response time of all nodes in the search tree, and the variability index is determined based on the response time of each node in the search tree and the average response time.

[0123] That is, in this embodiment of the invention, the weight coefficient of the calculation node load can be automatically optimized according to the actual operating conditions to adapt to the constantly changing data access patterns and optimize the accuracy and efficiency of load analysis.

[0124] In this embodiment of the invention, the aforementioned first weighting coefficient Second weighting coefficient The following formula can be used for optimization and updating:

[0125]

[0126]

[0127] in, The updated first weighting coefficient, The updated second weighting coefficients, This is the current first weighting coefficient. This is the current second weighting coefficient. To adjust the rate, the magnitude of the adjustment of the weighting coefficients (first weighting coefficient and second weighting coefficient) is controlled, and the sensitivity of the adjustment is controlled, so as to avoid system instability caused by over-adjustment. This is a performance evaluation function that adjusts the weighting coefficients based on system performance feedback, considering the direction and magnitude of the adjustment.

[0128] The performance evaluation function can be determined using the following formula:

[0129]

[0130] In the above formula, It can be or ,when for hour, for , for ,when for hour, for , for For ease of explanation, and Merging instructions.

[0131] in, This represents an efficiency metric that measures the data retrieval efficiency under the current weighting coefficients. Response time is a key indicator for measuring system performance. By optimizing the weighting coefficients to reduce the average response time, the overall efficiency of the system can be improved. The average response time refers to the average response time of all nodes in the entire search tree. This represents a variability index, measuring the fluctuation in system performance caused by adjustments to weighting coefficients. where n is the number of nodes in the search tree. For the first The response time of each node and excessive fluctuations in system performance can lead to a decline in user experience. By minimizing variability, system stability can be maintained when adjusting weighting coefficients.

[0132] In some embodiments, optionally, the load metric information includes: the number of child nodes of the node; wherein, determining whether to adjust the structure of the search tree based on the load metric information includes:

[0133] Step S141c: When the number of child nodes of the node exceeds the preset maximum number of child nodes, determine to adjust the structure of the search tree, wherein adjusting the structure of the search tree includes: splitting the node;

[0134] Step S142c: When the number of child nodes of the node is less than the preset minimum number of child nodes, determine to adjust the structure of the search tree, wherein adjusting the structure of the search tree includes: merging the node with the node's neighboring nodes.

[0135] In this embodiment of the invention, in order to maintain the balance of the search tree and query efficiency, when a node has too many or too few child nodes, it is necessary to perform node splitting or merging operations:

[0136]

[0137] in, This represents the operation performed on node n, which can be a split or merge operation. This represents the number of child nodes of node n. The `split` operation is triggered when the number of child nodes exceeds the maximum number of child nodes M. The split operation divides an overloaded node into multiple nodes, each carrying a portion of the original node's child nodes, thereby reducing the load on individual nodes and improving query efficiency. The `combine(n)` operation is triggered when the number of child nodes is less than the minimum number of child nodes m. The combinator merges underloaded nodes with their neighboring nodes (as children of the neighboring nodes), reducing the depth of the search tree and the total number of nodes, which helps reduce search paths and improve caching efficiency.

[0138] In this embodiment of the invention, plaintext data content can be persisted by creating multiple data objects for the plaintext data content. These data objects are mapped to mutable structured or binary objects via byte arrays. Copy-on-write (i.e., copying the original data object when it is modified) is used to maintain the management of encrypted data blocks and versions in the underlying storage. The data objects manage byte arrays by referencing data blocks through copy-on-write, starting from an empty sequence of data blocks, and supporting operations such as truncation, appending, and random access read / write.

[0139] That is, in some embodiments, optionally, it also includes:

[0140] Multiple data objects are defined for the plaintext data content, and the types of the data objects include root directory, directory, subdirectory, file and / or data;

[0141] An initial version object is created for each data object, and the version object is associated with multiple data blocks and multiple corresponding encrypted data blocks corresponding to the data object;

[0142] If the data object is modified, a new version object is created for the data object.

[0143] In this embodiment of the invention, the data object model supports the dynamic creation, modification, and version control of data, files, and directories, while data privacy is protected through encryption and access control mechanisms.

[0144] The data object contains the following information:

[0145] Object Identifier: Each data object has a unique identifier used to distinguish different storage objects.

[0146] Object type: Distinguishes whether the data object is structured data, a file, or a directory.

[0147] Data structure: For files, the data structure is usually a sequence of bytes; for directories, the data structure is a mapping that maps names to identifiers of their child objects.

[0148] Metadata: This includes the attributes of data objects, such as creation time, modification time, access time, permissions, etc.

[0149] Version control: Each data object may be associated with one or more versions, used to track historical changes to the data object.

[0150] Persistent references: The current state of a data object can reference a set of persistent data blocks.

[0151] Encryption and hashing: Data blocks are encrypted before storage, and a hash value of the encrypted data block is generated to ensure integrity and security.

[0152] In this embodiment of the invention, the data object can be initialized in the following manner:

[0153] Initializes an empty data object for new data, files, or directories and sets its initial state. The data object does not contain any data blocks in its initial state and has no data blocks associated with it. All operations are memory-based until the data blocks are persisted.

[0154] Allocate memory space for the data object to temporarily store the data that is about to be written. This memory space can be actual memory or a memory-mapped file.

[0155] Then, initialize the metadata of the data object, including file size, creation time, last modification time, etc.

[0156] Finally, the data objects are associated with a version control mechanism. Each data object is associated with an initial version object when it is created, and subsequent modifications will create new version objects.

[0157] Let's take a data object as an example for explanation.

[0158] Please refer to Figure 4 When the directory hierarchy is modified, a new block pointer is generated (each block pointer corresponds to a version object). The parent node index of the data object needs to be updated, and this update is then flushed to the root directory. To handle updates efficiently, each data object maintains an update object, which serves as a reference to its parent memory object.

[0159] During modification and persistence, the updated object is notified to associate with the new block pointer, and the parent object is synchronously modified to record the new version of the child object. All operations on the data tree are considered potential versioning operations, allowing the creation and sharing of new data snapshots.

[0160] Data versions can be named using block pointers to their version objects, which contain the data content and history. A directory list is a sequence of directory entries, each with a unique name mapped to a data object. Data persistence is achieved by serializing directory entries into version objects named by block pointers. Figure 4 An example of a directory hierarchy is shown. During persistence, the content of 'a' is stored in encrypted data blocks, and the block pointers of the version objects of the corresponding subdirectories are added to the version structure of 'a'. The version object of 'a' itself is also stored in encrypted data blocks, and the block pointers of the version objects of 'a' are stored in the parent directory entries of 'a'.

[0161] Persistent objects are initiated by specific triggering conditions, such as memory thresholds or user requests. To ensure the security of data block encryption, a unique content-based hash identifier is generated for each data block. A new version object is created using the data object's metadata, recording the current data block's state and metadata information. Simultaneously, the corresponding data block and version object are written to persistent storage, and the data system's index structure is updated to record these changes.

[0162] After the write operation is completed, data integrity checks are performed to ensure the accuracy of the data. Once it is confirmed that all data blocks and version objects have been successfully stored, the state of the data objects will be updated to be persistent, thus completing the entire persistence process.

[0163] In some embodiments, optionally, the method further includes:

[0164] A directed acyclic graph is constructed based on the version objects corresponding to the multiple data objects, wherein each node of the directed acyclic graph represents a version object of the data object, the edges of the directed acyclic graph represent the parent-child relationship between the data objects, and the information of the node includes the version ID;

[0165] When the data object is modified, a new node is created in the directed acyclic graph. The new node corresponds to the new version object of the data object and serves as a child node of the node of the initial version object of the data object. The information of the new node includes a version ID and a parent version ID, where the parent version ID is the version ID of the node of the initial version object of the data object.

[0166] In the embodiments of this invention, please refer to Figure 2 The organizational structure of data objects can be represented by a directed acyclic graph.

[0167] The key concepts of a directed acyclic graph are as follows:

[0168] 1. Node: Each node represents a data object. The necessary information for each node includes:

[0169] Node ID: A unique identifier for the node.

[0170] Node type: Identifies whether the node is data, a file, or a directory.

[0171] Content hash: If it is a file, it is the cryptographic hash value of the file content.

[0172] 2. Edge: An edge represents a parent-child relationship, that is, a relationship between data, files, or directories. It points from a node to its child nodes.

[0173] 3. Version Object: Each version object represents the state of the file system at a specific point in time. It contains:

[0174] Version ID, a unique identifier;

[0175] A root node reference, pointing to the root node of the directed acyclic graph, representing the file system structure of this version;

[0176] The parent version ID references the previous version object, forming a version chain.

[0177] 4. Root node: The starting point of a directed acyclic graph, representing the top-level directory of the data system.

[0178] 5. Child nodes: Each node can have multiple child nodes, representing data, files, and subdirectories in the directory.

[0179] 6. Immutability: Once a node is created, all its attributes (including content hash, etc.) cannot be changed. Any modification will result in the creation of a new node, not an alteration of the existing node.

[0180] like Figure 5 As shown, a version object consists of a set of immutable blocks (i.e., encrypted data blocks) of arbitrary length. Each block accesses data through a block pointer. Multiple version objects can reference the same underlying immutable block. The information of a version object is stored in encrypted data blocks and named using its encrypted hash value. The information includes block pointers to previous version objects, and it can manage any number of version objects containing immutable content. Ultimately, the data consistency problem is simplified to a version control problem: two data files may contain different blocks, but different version objects of the same block will not appear.

[0181] When a file or directory in the file system changes, a new version object is created. This new version object references the root node of the current directed acyclic graph and records the parent version ID.

[0182] By using the parent version ID of the version object, we can trace the historical changes of the file system and form a version chain.

[0183] Directed acyclic graphs allow branching and merging operations, allowing different version objects to evolve in parallel and eventually merge back into the initial version object.

[0184] The following example illustrates the data storage structure for directed acyclic graphs.

[0185] Assume the data storage structure of a directed acyclic graph (DAG) is as follows: Figure 6 As shown, each data and directory version object is a node in a directed acyclic graph. Each node contains the following information:

[0186] Node ID: A unique identifier.

[0187] Node type: data, file, or directory.

[0188] Data content: The content of data, files, or a list of child nodes of a directory.

[0189] Version information: A node link pointing to the previous version.

[0190] Figure 6 In the illustrated embodiment, the method for constructing the directed acyclic graph is as follows:

[0191] 1. Initialization: Create the root node, which represents the root directory of the data storage system.

[0192] 2. Add data: Create a data1 node under the root node, which contains the data content of data1 and version object information.

[0193] 3. Create a directory: Create a directory1 node under the root node, which contains a list of child nodes and version object information.

[0194] 4. Add a sub-file: Create a data2 node under the directory1 node, which contains the data content and version object information of data2.

[0195] Please refer to Figure 7 The version control method is as follows:

[0196] 1. When modifying data1, create a new data1 node (data1(v2)) containing the updated data content (content of data1-v2) and a version link pointing to the original data1 node (data1(v1)).

[0197] 2. When modifying directory1 or its child nodes, the same version control logic is followed, creating new nodes and retaining version history.

[0198] In some embodiments, the method may optionally further include: sharing the encrypted data block with the data manager as a data provider.

[0199] Existing technological solutions often lack effective incentive mechanisms to promote cooperation between data managers and data consumers regarding the protection and sharing of privacy-related data. This results in the underutilization of the potential value of data and a lack of effective protection for the privacy rights of data owners. Different data owners may have different needs and standards for privacy protection, and existing technological solutions often adopt a "one-size-fits-all" approach to privacy protection, failing to fully consider this heterogeneity.

[0200] In this embodiment of the invention, by introducing a market incentive mechanism, users can make an informed trade-off between protecting their personal privacy and obtaining value incentives. This not only resolves the conflict between privacy protection and data value but also enhances users' awareness and control over the value of their data by providing a transparent value return mechanism. The level of privacy protection can be flexibly adjusted according to changes in personal privacy data classification requirements and the data circulation market environment. This dynamic adjustment capability strengthens the personalization and adaptability of privacy protection measures, responding promptly to changes in user needs, thereby improving user satisfaction and trust in data sharing and circulation.

[0201] Please refer to Figure 8 This invention also provides a data processing method, including:

[0202] Step S21: Provide the encrypted data block provided by the data provider to the data consumer through the data circulation network. The encrypted data block is obtained and stored by the data provider through the following method: dividing the plaintext data content into multiple data blocks and determining the block pointer of each data block. The block pointer of the data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block. The encrypted data block is obtained by encrypting the data block using the encryption key.

[0203] In the embodiments of this invention, please refer to Figure 9 This paper proposes four key entities for privacy data circulation scenarios: data managers, data owners, data circulation networks, and data consumers. Data owners, as data providers, securely store data using the aforementioned scheme. Data managers register with the data circulation network and apply to the data owners for management rights of their privacy data. Data consumers, as demanders in the privacy data circulation market, obtain privacy data from the secure storage system through the data circulation network. Rapid data circulation can be achieved by sharing block pointers of data blocks in the above embodiments, without copying or moving the data itself.

[0204] To ensure that the flow of data among these four entities strikes a balance between protecting data privacy and maximizing data value, participants in the data flow market are required to jointly comply with transparent data use policies and privacy protection standards.

[0205] The definitions of these four data flow model entities are as follows:

[0206] 1. Data Manager (DM):

[0207] A data manager is an entity that can control data by providing centralized storage services for user data and processing and analyzing this data to improve service quality or for commercial use.

[0208] In the market for privacy-preserving data, data managers act as data providers. They determine the amount and type of data that can be circulated based on tiered privacy requirements and market incentive mechanisms.

[0209] 2. Data Owner (DO):

[0210] Data owners are individuals or groups who use services provided by data managers and generate or store personal privacy data during the use of these services, including: personal data files, personal information (such as location information, usage records, data browsing habits), etc.

[0211] Data owners have decision-making power in the privacy data circulation market, and can authorize data managers to dispose of their data and profit from it.

[0212] 3. Data Networks (DN):

[0213] The data circulation network acts as an intermediary, connecting data managers and data owners. It acquires data from data managers and uses it for data value calculation, data circulation effect tracking, and other purposes.

[0214] The role of data circulation networks is to optimize the efficiency of data circulation and improve the relevance and conversion rate of data. At the same time, it is necessary to ensure compliance with privacy regulations and the tiered privacy requirements of data owners.

[0215] 4. Data Consumer (DC):

[0216] Data consumers are business entities that use data products or services and further integrate data obtained from data circulation networks to facilitate the provision of combined and comprehensive data service capabilities.

[0217] In the privacy data circulation market, data consumers, as users of the data, pay fees to data managers according to market mechanisms in order to obtain the privacy data of the data owners.

[0218] The circulation of privacy data can be categorized into competitive markets and oligopolistic markets based on market structure. A competitive market refers to an environment with numerous data managers offering similar or homogeneous data, leading to convergent prices and circulation conditions. Conversely, an oligopolistic market is dominated by a few data managers, each potentially possessing unique or differentiated privacy data, thus wielding greater bargaining power in the market.

[0219] In a competitive market, the goal of data managers is to maximize the benefits of data circulation within a given market profit level. This market structure encourages data managers to attract data consumers by providing high-quality data and data circulation mechanisms, thereby gaining more benefits in the face of fierce market competition.

[0220] Unlike competitive markets, oligopolistic markets are characterized by strategic interactions among data managers who adjust the mechanisms and volume of privacy data circulation based on market conditions and competitors' behavior in order to achieve their optimal position in the market.

[0221] In a competitive market, market equilibrium can often maximize the value of data applications; however, in an oligopolistic market, value maximization may not be achieved because data managers may use strategic behavior to obtain higher profits.

[0222] In some embodiments, optionally, the method further includes:

[0223] The data manager constructs a supply function, which describes the amount of data the data manager is willing to provide for the benefit gained per unit of data circulation;

[0224] The data manager provides the supply function to the data circulation network, which then determines the optimal benefit the data manager receives from the data circulation network for each unit of data circulation based on the supply function.

[0225] In this embodiment of the invention, the data manager defines the amount of privacy data allowed to circulate for specific interests through a supply function. The supply function is a function of the amount of privacy data circulated and the benefit gained per unit of data circulated, reflecting the data manager's tiered privacy mechanism and market strategy. The supply function pricing mechanism allows the data manager to adjust its privacy data circulation strategy based on market conditions and the goal of maximizing data value, while protecting the privacy of data owners.

[0226] The process of supply function pricing will be explained in detail below:

[0227] Step 1: Constructing and setting parameters for the supply function.

[0228] Data managers (DMs) construct supply functions to describe the amount of privacy data circulating for specific interests. Each DM must adhere to privacy requirements derived from the privacy tiers set by its data owners. The supply function is a function of the amount of privacy data circulating and the benefit gained per unit of data circulation, parameterized as follows: ,in This is the bidding parameter. The function below reflects the data manager's privacy preferences and market strategy, and is the core of the bidding mechanism.

[0229]

[0230] Supply function This indicates that data manager i is willing to share the benefits gained from the circulation of each unit of privacy data. The amount of private data provided. N represents the number of data management parties.

[0231] Non-negative pricing parameters This reflects the data manager's sensitivity or willingness regarding the flow of privacy-sensitive data. In an oligopolistic market, the data manager can adjust... To implement strategic behaviors, such as by increasing or decreasing This can influence market clearing prices, thereby gaining an advantage in the competition.

[0232] This linear function simplifies bidding strategies, allowing data administrators to adjust their strategies based on their privacy tiering policies and market conditions. The amount of private data a data manager is willing to provide is the product of their bidding parameters and the benefit gained per unit of data circulation.

[0233] Step 2: Market liquidation and profit distribution.

[0234] The data circulation network acts as an intermediary, determining the market clearing price—the profit earned by the data manager (DM) for each unit of privacy data circulation—based on the bidding results of the supply function. This process requires considering the bids from all data managers and the total demand from data consumers. The goal of market clearing is to find a price that equals total supply to total demand, while maximizing the value of privacy data.

[0235]

[0236] in, The market clearing price represents the optimal benefit that the data manager obtains from the data circulation network for each unit of privacy data circulation.

[0237] D represents the total demand for privacy data circulation obtained by the data circulation network from data consumers.

[0238] For data manager i at a given market liquidation price The amount of private data that the client is willing to provide.

[0239] The total amount of privacy data that all data managers are willing to provide.

[0240] Market clearing price The price is determined inversely by the total demand (d) and the total supply that all data managers are willing to provide at that price. By providing a dynamic adjustment mechanism, the data circulation market allows prices to be adjusted according to real-time supply and demand conditions. This flexibility helps the market respond quickly to external changes. This price ensures market supply and demand balance while maximizing the value of privacy-preserving data.

[0241] Step 3: Decision-making on strategies for the circulation of privacy data.

[0242] When developing supply functions, data managers need to consider privacy requirements, market conditions, and consumer data demands. They must find a balance between protecting consumer privacy and maximizing benefits. This typically involves cost-benefit analysis and forecasting market dynamics.

[0243]

[0244] in, The revenue for the data manager is the total benefit derived from the data circulating in the market minus the costs.

[0245] The market clearing price is the profit that the data manager receives from the data circulation network for each unit of privacy data circulation.

[0246] The amount of private data that the data management party is willing to circulate.

[0247] i provides data management services Costs incurred due to the privacy data of organizations.

[0248] This formula states that the data manager's goal is to maximize their net benefit, which is the revenue gained from the market minus the cost of providing the data. This is achieved by considering the market settlement price. The data management party's strategic decisions are able to adapt to market changes. This reflects the decision-making process by which the data management party seeks a balance between privacy protection and data value.

[0249] Step 4: Submission of supply function bids and market response.

[0250] Data managers submit corresponding supply functions using a data circulation network, which then adjusts its market strategies based on these bids, including profit distribution and privacy data tiering strategies. This process is dynamic because market conditions and participant strategies may change over time. The data circulation network needs to monitor market dynamics in real time and adjust its clearing strategies as needed.

[0251]

[0252] in, The optimal amount of privacy data that data manager i submits to the market, i.e., the market settlement price for data manager i. The maximum amount of private data that is willing to be provided.

[0253] The revenue for the data management party is determined by market prices. Privacy data volume The cost of privacy-related data circulation Decision made, see step 4.

[0254] Data management party through This adjustment to its market share and revenue emphasizes the balance between privacy protection and data value. Data managers, in deciding... In this process, the costs and potential benefits of privacy protection must be considered to find the optimal balance between protecting privacy and generating revenue. Data with higher privacy metrics may require higher market prices to compensate for privacy risks, thus influencing the bidding strategies and market responses of data managers. During market clearing, the data circulation network needs to balance the interests of all parties to ensure that the pricing of the data manager's supply function matches the demand of data consumers, while maximizing the overall market value of data applications.

[0255] Supply-function pricing mechanisms emphasize finding a balance between privacy protection and market efficiency. Data managers need to consider consumer privacy settings and legal requirements when setting prices to ensure the legality of data circulation. Through reasonable market design and mechanism arrangements, supply-function pricing mechanisms can promote the transparency and fairness of data circulation, improve market efficiency, and protect consumer privacy rights.

[0256] Please refer to Figure 10 This invention also provides a data processing apparatus 100, comprising:

[0257] The encryption module 101 is used to divide plaintext data content into multiple data blocks and determine the block pointer of each data block. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key.

[0258] Storage module 102 is used to store each of the encrypted data blocks and the corresponding block pointers;

[0259] Search tree creation module 103 is used to create a search tree, wherein each node of the search tree corresponds to one of the encrypted data blocks, and the information of the node includes: the name of the encrypted data block, a pointer to the storage location of the encrypted data block, and pointers to the child nodes of the node; the name of the encrypted data block is the hash value of the data block;

[0260] The search tree adjustment module 104 is used to obtain the load metric information of each node of the search tree and determine whether to adjust the structure of the search tree based on the load metric information.

[0261] Optionally, the load metric information includes: the load of the node at the next time point;

[0262] The search tree adjustment module 104 is used for:

[0263] The load of the node at the current time is determined based on the access frequency of the node at the current time and the first weight coefficient corresponding to the access frequency, the number of child nodes of the node at the current time and the second weight coefficient corresponding to the number of child nodes.

[0264] Based on the node's load at the current time and its historical load, predict the node's load at the next time.

[0265] If the load of the node at the next time point is greater than or equal to a first threshold, it is determined to adjust the structure of the search tree, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, reconstructing the search tree, or splitting the node.

[0266] Optionally, the load metric information includes: the load difference between the node and its neighboring nodes;

[0267] The search tree adjustment module 104 is used for:

[0268] The load of a node is determined based on the node’s access frequency and the first weight coefficient corresponding to the access frequency, the number of the node’s child nodes and the second weight coefficient corresponding to the number of child nodes.

[0269] The load of the adjacent nodes is determined based on the access frequency of the adjacent nodes and the first weight coefficient corresponding to the access frequency, as well as the number of child nodes of the adjacent nodes and the second weight coefficient corresponding to the number of child nodes.

[0270] Calculate the load difference between the node and its neighboring nodes based on the node's load and the load of the neighboring nodes;

[0271] If the load difference between the node and its neighboring nodes is greater than or equal to a second threshold, the structure of the search tree is determined to be adjusted, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, or reconstructing the search tree.

[0272] Optional, also includes:

[0273] An update module is used to update the first weight coefficient and the second weight coefficient according to a preset adjustment rate and a performance evaluation function. The performance evaluation function is determined based on an efficiency index and a variability index. The efficiency index is determined based on the average response time of all nodes in the search tree, and the variability index is determined based on the response time of each node in the search tree and the average response time.

[0274] Optionally, the load metric information includes: the number of child nodes of the node;

[0275] The search tree adjustment module 104 is used for:

[0276] When the number of child nodes of a node exceeds a preset maximum number of child nodes, the structure of the search tree is adjusted, wherein adjusting the structure of the search tree includes splitting the node;

[0277] When the number of child nodes of a node is less than a preset minimum number of child nodes, the structure of the search tree is adjusted, wherein adjusting the structure of the search tree includes merging the node with its neighboring nodes.

[0278] Optional, also includes:

[0279] The definition module is used to define multiple data objects for the plaintext data content. The types of the data objects include root directory, directory, subdirectory, file and / or data.

[0280] The first creation module is used to create an initial version object for each data object, wherein the version object is associated with multiple data blocks and multiple corresponding encrypted data blocks corresponding to the data object;

[0281] The second creation module is used to create a new version object for the data object if the data object is modified.

[0282] Optional, also includes:

[0283] The third creation module is used to construct a directed acyclic graph based on the version objects corresponding to the multiple data objects. Each node of the directed acyclic graph represents a version object of the data object, the edges of the directed acyclic graph represent the parent-child relationship between the data objects, and the information of the node includes the version ID.

[0284] The fourth creation module is used to create a new node in the directed acyclic graph when the data object is modified. The new node corresponds to a new version object of the data object and serves as a child node of the node of the initial version object of the data object. The information of the new node includes a version ID and a parent version ID, where the parent version ID is the version ID of the node of the initial version object of the data object.

[0285] Optional, also includes:

[0286] A sharing module is used to share the encrypted data block with the data management party as a data provider.

[0287] Please refer to Figure 11 This invention also provides a data processing apparatus 200, comprising:

[0288] The management module 201 is used to provide encrypted data blocks provided by the data provider to the data consumer through a data circulation network. The encrypted data blocks are obtained and stored by the data provider through the following method: dividing plaintext data content into multiple data blocks and determining a block pointer for each data block. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key.

[0289] Optional, also includes:

[0290] A construction module is used to construct a supply function, which describes the amount of data that the data manager is willing to provide for the benefit obtained per unit of data circulation;

[0291] A module is provided to provide the supply function to the data circulation network, and the data circulation network determines the optimal benefit that the data manager can obtain from the data circulation network for each unit of data circulation based on the supply function.

[0292] Please refer to Figure 12 The present invention also provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the computer program is executed by the processor 301, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0293] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0294] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 or Figure 8 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0295] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0296] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0297] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, include: The plaintext data content is divided into multiple data blocks, and a block pointer for each data block is determined. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key. Store each of the encrypted data blocks and the corresponding block pointers; Create a search tree, where each node of the search tree corresponds to one of the encrypted data blocks. The information of the node includes: the name of the encrypted data block, a pointer to the storage location of the encrypted data block, and pointers to the child nodes of the node; the name of the encrypted data block is the hash value of the data block. Obtain the load metric information of each node in the search tree, and determine whether to adjust the structure of the search tree based on the load metric information.

2. The method according to claim 1, characterized in that, The load metric information includes: the load of the node at the next time point; The process of obtaining load metric information for each node of the search tree includes: The load of the node at the current time is determined based on the access frequency of the node at the current time and the first weight coefficient corresponding to the access frequency, the number of child nodes of the node at the current time and the second weight coefficient corresponding to the number of child nodes. Based on the node's load at the current time and its historical load, predict the node's load at the next time. The process of determining whether to adjust the structure of the search tree based on the load metric information includes: If the load of the node at the next time point is greater than or equal to a first threshold, it is determined to adjust the structure of the search tree, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, reconstructing the search tree, or splitting the node.

3. The method according to claim 1, characterized in that, The load metric information includes: the load difference between the node and its neighboring nodes; The process of obtaining load metric information for each node of the search tree includes: The load of a node is determined based on the node’s access frequency and the first weight coefficient corresponding to the access frequency, the number of the node’s child nodes and the second weight coefficient corresponding to the number of child nodes. The load of the adjacent nodes is determined based on the access frequency of the adjacent nodes and the first weight coefficient corresponding to the access frequency, as well as the number of child nodes of the adjacent nodes and the second weight coefficient corresponding to the number of child nodes. Calculate the load difference between the node and its neighboring nodes based on the node's load and the load of the neighboring nodes; The process of determining whether to adjust the structure of the search tree based on the load metric information includes: If the load difference between the node and its neighboring nodes is greater than or equal to a second threshold, the structure of the search tree is determined to be adjusted, wherein adjusting the structure of the search tree includes at least one of the following: performing a local tree rotation on the search tree, or reconstructing the search tree.

4. The method according to claim 2 or 3, characterized in that, Also includes: The first weight coefficient and the second weight coefficient are updated according to the preset adjustment rate and performance evaluation function. The performance evaluation function is determined based on the efficiency index and the variability index. The efficiency index is determined based on the average response time of all nodes in the search tree, and the variability index is determined based on the response time of each node in the search tree and the average response time.

5. The method according to claim 1, characterized in that, The load metric information includes: the number of child nodes of the node; The process of determining whether to adjust the structure of the search tree based on the load metric information includes: When the number of child nodes of a node exceeds a preset maximum number of child nodes, the structure of the search tree is adjusted, wherein adjusting the structure of the search tree includes splitting the node; When the number of child nodes of a node is less than a preset minimum number of child nodes, the structure of the search tree is adjusted, wherein adjusting the structure of the search tree includes merging the node with its neighboring nodes.

6. The method according to claim 1, characterized in that, Also includes: Multiple data objects are defined for the plaintext data content, and the types of the data objects include root directory, directory, subdirectory, file and / or data; An initial version object is created for each data object, and the version object is associated with multiple data blocks and multiple corresponding encrypted data blocks corresponding to the data object; If the data object is modified, a new version object is created for the data object.

7. The method according to claim 6, characterized in that, Also includes: A directed acyclic graph is constructed based on the version objects corresponding to the multiple data objects, wherein each node of the directed acyclic graph represents a version object of the data object, the edges of the directed acyclic graph represent the parent-child relationship between the data objects, and the information of the node includes the version ID; When the data object is modified, a new node is created in the directed acyclic graph. The new node corresponds to the new version object of the data object and serves as a child node of the node of the initial version object of the data object. The information of the new node includes a version ID and a parent version ID, where the parent version ID is the version ID of the node of the initial version object of the data object.

8. The method according to claim 1, characterized in that, Also includes: The encrypted data block is shared between the data provider and the data manager.

9. A data processing method, characterized in that, include: Encrypted data blocks provided by the data provider are provided to the data consumer through a data circulation network. The encrypted data blocks are obtained and stored by the data provider through the following method: dividing the plaintext data content into multiple data blocks and determining the block pointer of each data block. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block. The encrypted data block is obtained by encrypting the data block using the encryption key.

10. The method according to claim 9, characterized in that, Also includes: Construct a supply function that describes the amount of data that the data manager is willing to provide for the benefit gained from each unit of data circulation; The supply function is provided to the data circulation network, which then determines the optimal benefit that the data manager can obtain from the data circulation network for each unit of data circulation based on the supply function.

11. A data processing apparatus, characterized in that, include: An encryption module is used to divide plaintext data content into multiple data blocks and determine a block pointer for each data block. The block pointer of each data block includes: an encryption key for the data block and a hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key. The storage module is used to store each of the encrypted data blocks and the corresponding block pointers; The search tree creation module is used to create a search tree, where each node of the search tree corresponds to one of the encrypted data blocks. The information of the node includes: the name of the encrypted data block, a pointer to the storage location of the encrypted data block, and pointers to the child nodes of the node; the name of the encrypted data block is the hash value of the data block. The search tree adjustment module is used to obtain the load metric information of each node of the search tree, and determine whether to adjust the structure of the search tree based on the load metric information.

12. A data processing apparatus, characterized in that, include: The management module is used to provide encrypted data blocks provided by the data provider to the data consumer through a data circulation network. The encrypted data blocks are obtained and stored by the data provider through the following method: dividing plaintext data content into multiple data blocks and determining a block pointer for each data block. The block pointer of each data block includes: the encryption key of the data block and the hash value of the encrypted data block. The encryption key is the hash value of the data block, and the encrypted data block is obtained by encrypting the data block using the encryption key.

13. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 10.

Images