Distributed storage method and system for data security

Abstract

The invention discloses a distributed storage method and system for data security, and belongs to the technical field of data storage security, the method comprises the following steps: firstly, fragmenting original data, and generating a unique identifier for each data fragment; for each fragment, an encryption algorithm and key credentials are adaptively selected, and the key credentials are stored in a key management system. By performing homomorphic encryption on the key voucher, an encryption handle is generated, and efficient positioning and retrieval of the key are realized. According to the method, a differentiated encryption strategy is adopted, a most suitable encryption scheme is selected according to fragment characteristics, and encrypted data fragments are dispersed and stored in a plurality of distributed storage nodes. A global mapping index is constructed in a metadata management server, and a corresponding relation between an original data global identifier and metadata of each data fragment is recorded, including information such as identifiers before and after encryption, a storage position, an encryption algorithm, a key handle and the like. According to the method, the data leakage risk is effectively reduced, and the safety and reliability of the storage system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage security, and in particular to a distributed storage method and system for data security. Background Art

[0002] With the advent of the cloud computing and big data era, data storage and sharing have become important needs for enterprises and organizations. Traditional centralized storage solutions have serious problems such as high risk of single point failure, weak data security, and difficult data recovery. Although distributed storage technology has emerged, existing technologies still face technical challenges such as data sharding encryption, key management, and data integrity tracing in the multi-node storage process.

[0003] Existing distributed storage solutions usually use simple encryption methods, which cannot effectively meet the differentiated security requirements of different data shards. Traditional methods have complexity and security issues in key management and shard positioning, making it difficult to achieve efficient and secure data storage and retrieval.

[0004] Data security has become a key issue in the current development of information technology. Existing technologies lack a comprehensive technical solution that can simultaneously meet the requirements of data sharding refined encryption, efficient key management, and flexible storage positioning in a multi-node distributed storage environment. This not only restricts the security of distributed storage systems, but also limits the efficient and secure sharing and use of large-scale data. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a distributed storage method and system for data security to solve the above technical problems.

[0006] To achieve the above object, in a first aspect, a distributed storage method for data security is provided, which comprises the following steps: Slice the original data to be stored to generate multiple data slices, and determine the global identifier of the original data and the pre-encryption slice identifier of each data slice having a slice sequence number; For each of the data slices, respectively determine a corresponding adaptive encryption algorithm and key certificate, and store the key certificate of each data slice in a key management system; Performing homomorphic encryption processing on the key certificate of each data shard, and using the obtained encryption result as the encryption handle of the key certificate, wherein the encryption handle is used to locate and retrieve the corresponding key certificate in the key management system; According to the encryption algorithm and key certificate corresponding to each data slice, each data slice is encrypted to obtain a plurality of encrypted data slices, and an encrypted slice identifier having a slice sequence number is determined for each encrypted data slice; According to the multi-dimensional distributed storage rule, the plurality of encrypted data shards are dispersedly stored in a plurality of distributed storage nodes, and the location information of each of the encrypted data shards on the corresponding distributed storage node is recorded; A global mapping index indicating the correspondence between the global identifier of the original data and the metadata information of each of the data shards is stored in a metadata management server; the metadata information of each of the data shards includes: the shard identifier before encryption, the shard identifier after encryption, the location information, the encryption algorithm, and the encryption handle of the key certificate corresponding to the data shard.

[0007] In a second aspect, a distributed storage system for data security is provided, comprising: The data sharding module is used to perform sharding processing on the original data to be stored, generate multiple data shards, and determine the global identifier of the original data and the shard identifier before encryption of each data shard; An encryption algorithm selection module is used to adaptively determine the encryption algorithm and key credentials for each data shard, and store the key credentials in a key management system; A key encryption module, used to perform homomorphic encryption on the key credential to generate an encryption handle, wherein the encryption handle is used to locate and retrieve the corresponding key credential in the key management system; A data shard encryption module, used to encrypt the data shards according to the selected encryption algorithm and key certificate, generate encrypted data shards, and determine the encrypted shard identifier of each encrypted data shard; A distributed storage scheduling module, used to disperse and store the encrypted data shards to multiple distributed storage nodes according to multi-dimensional storage rules, and record the location information of each encrypted data shard; A metadata management server is used to store a global mapping index indicating the correspondence between the global identifier of the original data and the metadata information of each data shard, wherein the metadata information includes: the shard identifier of the data shard before encryption, the shard identifier after encryption, the location information, the encryption algorithm and the encryption handle of the key certificate.

[0008] The above technical solution has the following beneficial technical effects: The distributed storage method provided by the embodiment of the present invention improves data storage security and system reliability by realizing adaptive encryption, homomorphic encryption key management and multi-dimensional distributed storage of data shards. The method realizes differentiated encryption processing of data shards, dynamically selects the most suitable encryption algorithm and key according to the characteristics of each shard, and effectively improves the accuracy of data security protection; by homomorphically encrypting the key credentials and generating encryption handles, a set of efficient and secure key management mechanisms is established, which enhances the convenience of key retrieval and positioning; adopts multi-dimensional distributed storage rules to make data shards dispersed and stored in multiple nodes, effectively reducing the risk of single point failure; at the same time, a global mapping index is constructed in the metadata management server, which realizes the accurate tracking and rapid reconstruction of data shards, which not only ensures the integrity and traceability of the data, but also improves the overall performance and security of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Figure 1 is a flow chart of a distributed storage method for data security according to an embodiment of the present invention; Figure 2 is a flowchart of step S10 of an embodiment of the present invention; Figure 3 is a flowchart of step S20 of an embodiment of the present invention; Figure 4 is a flowchart of step S30 of an embodiment of the present invention; Figure 5 is a flowchart of step S40 of an embodiment of the present invention; Figure 6 is a flowchart of step S50 of an embodiment of the present invention; Figure 7 is a flowchart of step S52 of an embodiment of the present invention; Figure 8 is a flowchart of step S60 of an embodiment of the present invention; Fig. 9 is a flow chart of another distributed storage method for data security according to an embodiment of the present invention; Fig.10 is a functional block diagram of a distributed storage system for data security according to an embodiment of the present invention; Fig.11 It is a schematic diagram of the structure of a computer system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0010] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0011] Embodiment 1 like Figure 1 As shown, this embodiment provides a distributed storage method for data security, which is performed by a distributed storage system for data security and includes the following steps: S10: Slice the original data to be stored to generate multiple data slices, and determine the global identifier of the original data and the pre-encryption slice identifier of each data slice having a slice sequence number.

[0012] Specifically, for raw data of different types and sizes, this step adopts an intelligent sharding strategy. For example, for a large file, the system first calculates its globally unique identifier, and can use the keywords or subject terms and timestamps of the raw data as global identifiers. The sharding process is based on the characteristics of the file itself, such as file type, file size, and content structure. The shards can be equal-length shards or content-based dynamic shards. The system divides the file corresponding to the raw data into multiple data shards based on preset rules and file characteristics. Each data shard will obtain a unique pre-encryption shard identifier, which consists of the original file hash value and the shard sequence number to ensure the uniqueness and traceability of each data shard.

[0013] S20: For each of the data slices, determine a corresponding adaptive encryption algorithm and key certificate, and store the key certificate of each data slice in a key management system.

[0014] Specifically, the most suitable encryption scheme is dynamically selected according to the characteristics and sensitivity of each data shard. Reference factors for encryption algorithm selection include shard size, data type, expected security level, etc. For example, different encryption algorithms can be selected for shards of different sizes and types, such as AES (Advanced Encryption Standard)-128 for small text shards and AES-256 for large binary files. The key credential contains not only the type of encryption algorithm, but also meta-information such as key length, generation time, and usage restrictions. The system generates a unique and strongly random encryption key for each shard and stores it in a dedicated key management system through a secure channel. The key management system uses multi-layer encryption and strict access control mechanisms to ensure the absolute security of the key.

[0015] S30: Perform homomorphic encryption processing on the key certificate of each data shard, and use the obtained encryption result as the encryption handle of the key certificate, which is used to locate and retrieve the corresponding key certificate in the key management system.

[0016] Specifically, the system uses a homomorphic encryption algorithm (such as Paillier homomorphic encryption algorithm or improved RSA homomorphic encryption) to encrypt the key credentials of each data shard. Homomorphic encryption allows specific computing operations to be performed on data in a ciphertext state without decryption. The generated encryption handle can not only uniquely identify and locate the key credential in the key management system, but also perform necessary verification and management without fully exposing the key information. The encryption handle is a complex encrypted string that contains an encrypted representation of information such as the shard identifier and encryption algorithm, ensuring that even if the handle is intercepted, key information cannot be directly obtained.

[0017] S40: Encrypt each of the data slices according to the encryption algorithm and key certificate corresponding to each of the data slices to obtain a plurality of encrypted data slices, and determine an encrypted slice identifier having a slice sequence number for each of the encrypted data slices.

[0018] Specifically, for each data shard, the system performs precise encryption based on the previously determined adaptive encryption algorithm and unique key. The encryption mode can choose different working modes such as GCM (Gaussian Counter Mode) and CBC (Cipher Block Chaining), and cooperate with the randomly generated initialization vector to increase the randomness and security of encryption. Each encrypted shard will obtain a new encrypted shard identifier, which contains the original shard information and encryption mark. The system can also add additional random salt values ​​during the encryption process to further improve the unpredictability of encryption.

[0019] S50: According to the multi-dimensional distributed storage rule, the plurality of encrypted data shards are dispersedly stored in a plurality of distributed storage nodes, and the location information of each of the encrypted data shards on the corresponding distributed storage node is recorded.

[0020] Specifically, the multi-dimensional distributed storage rules consider multiple dimensions such as geographical distribution, node type diversity, load balancing and redundant backup. The selection and allocation of storage nodes are based on factors such as the availability, performance, and security level of the storage nodes. The location information record will include detailed information such as the node IP address, specific storage path, and storage timestamp. By distributing encrypted shards for storage, the security, reliability, and access performance of the data are improved.

[0021] S60: Store a global mapping index indicating the correspondence between the global identifier of the original data and the metadata information of each of the data shards in a metadata management server; the metadata information of each of the data shards includes: the shard identifier before encryption, the shard identifier after encryption, the location information, the encryption algorithm, and the encryption handle of the key certificate corresponding to the data shard.

[0022] Specifically, the global mapping index establishes the correspondence between the global identifier of the original data and its shard metadata, and stores it in the metadata management server. The metadata information of each data shard is a composite structure, which mainly includes five key elements: first, the shard identifier before encryption of the data shard, which is generated by combining the shard sequence number and a random number to ensure the uniqueness of each shard; second, the shard identifier after encryption, which is used to mark the encryption status of the shard; third, detailed location information, which records the specific node IP address, storage path, timestamp, etc. of the shard storage; fourth, encryption algorithm information, which can include technical details such as algorithm type, key length, encryption mode, etc.; finally, the encryption handle of the key credential, which is a special identifier generated by encryption technology, which can achieve secure positioning and retrieval of the corresponding key without directly exposing the key.

[0023] This embodiment uses an adaptive encryption algorithm and independent key credentials for each data shard to build a dynamic, multi-level data security protection system. By centrally storing key credentials in a dedicated key management system and performing homomorphic encryption on them, the high confidentiality and security of key information is achieved. This method not only enhances the flexibility of data encryption, but also effectively prevents the security risks faced by traditional centralized key management.

[0024] This embodiment designs a global mapping index mechanism and establishes an accurate correspondence between the global identifier of the original data and the metadata of each data shard. Through multi-dimensional distributed storage rules, the encrypted data shards are dispersed and stored in multiple storage nodes, and the complex mapping relationship is centrally maintained in the metadata management server, achieving high availability, high reliability and strong security of data storage. This architecture not only provides fine-grained data tracking capabilities, but also improves the fault tolerance and scalability of the entire distributed storage system.

[0025] This embodiment builds a comprehensive, multi-level data security protection system by embedding security mechanisms in various links such as data sharding, encryption, storage, and indexing. The encryption handle design makes the key retrieval process more secure and efficient, avoiding direct exposure of sensitive key information. At the same time, the application of shard storage and adaptive encryption algorithms not only maximizes data security, but also balances security and system performance to a certain extent.

[0026] like Figure 2 As shown, step S10 specifically includes the following steps: S11: extracting features from the original data, and analyzing the extracted features to obtain the type code, sensitivity level and information entropy of the original data; determining a feature vector according to the type code, the sensitivity level and the information entropy.

[0027] Specifically, the system first divides the raw data into basic types such as text, images, videos, tables, and binary through rule matching, and generates type codes that are a combination of numbers and letters. The sensitivity assessment uses predefined rules to divide the raw data into four levels: public data, internal data, restricted data, and highly sensitive data, mainly considering dimensions such as whether it contains personal information, whether it involves commercial secrets, and financial data. The information entropy calculation uses the classic Shannon entropy formula to measure the randomness and information density of the raw data, and the entropy value range is controlled between 0 and 8. Finally, the system converts the type code, sensitivity level, and information entropy into a 64-dimensional feature vector through a linear weighted mapping method.

[0028] The linear weighted mapping method is a technique for converting different feature dimensions into a unified feature vector. The specific implementation is to assign different weight coefficients to each feature dimension (type code, sensitivity level, information entropy), and then perform linear combinations. For example, assuming that the weight vector W = [w1, w2, w3], the feature vector X = [type code, sensitivity level, information entropy], then the mapping formula is: Feature vector = W1 Type code + W2 Sensitivity Level + W3 Information entropy. The weight coefficient is determined through experience or training data to ensure that the contribution of different dimensions to the final feature vector is consistent with the actual application scenario. This method can map heterogeneous features to a unified vector space and provide standardized input for subsequent deep learning tasks.

[0029] The rule matching method is a data type recognition technology based on predefined patterns and features. The system pre-sets a set of detailed rules and feature templates for identifying different types of data. For example, for text types, character encoding, language features, and text structure can be checked; for images, file extensions, pixel information, and color space are analyzed; for tabular data, row and column structures and data consistency are checked. The matching process is a step-by-step screening and comparison process that ultimately divides the data into the most matching type. This method is interpretable and flexible, and the matching rules can be quickly adjusted and optimized according to specific application scenarios.

[0030] S12: Using a pre-trained sharding deep learning model, dynamically determining sharding parameters according to the feature vector, sharding the original data according to the sharding parameters, and generating multiple data shards.

[0031] Specifically, the system inputs the feature vector generated in the previous stage into the pre-trained sharding deep learning model and outputs the corresponding sharding parameters. The sharding parameters mainly include: sharding size or sharding granularity, number of shards and sharding boundary conditions. The specific contents include: maximum sharding size limit, which is set according to the storage system and network transmission capacity; minimum sharding size, which ensures the effectiveness of sharding; sharding number range, which is used to balance storage efficiency and access performance; sharding boundary conditions, which can be semantic block sharding or mixed sharding mode; special sharding rules for special data types, such as sharding video files by key frames.

[0032] Information entropy is an important indicator for measuring data complexity and randomness. High information entropy indicates that the data contains more unpredictable information and complex structures, so a more fine-grained sharding strategy is needed. For data with high information entropy (such as encrypted files, complex images, or multimedia content), using smaller shards (1-10MB) can improve the flexibility of data transmission and processing and reduce the information redundancy of a single shard. In contrast, data with low information entropy (such as plain text documents and simple tables) can use larger shards (10-50MB) to reduce metadata overhead. This dynamic shard resizing mechanism based on information entropy not only optimizes storage efficiency, but also provides a finer control granularity for subsequent data reorganization and transmission.

[0033] Slicing boundary conditions refer to the rules and methods for determining how to divide and locate slicing in the original data. They mainly include the following dimensions: fixed-length boundaries, that is, evenly slicing according to the preset data length; semantic boundaries, segmenting according to the logical structure and intrinsic semantic features of the data, such as text by paragraphs, images by regions, and videos by key frames; adaptive boundaries, dynamically adjusting the slicing position to avoid slicing at key information or structural boundaries. Different boundary conditions are suitable for different data types and application scenarios. For example, text files can be divided by paragraphs or chapters, while binary files need to be sliced ​​by fixed length or specific tags. The choice of boundary conditions directly affects the integrity, readability, and reorganization efficiency of the data.

[0034] Hybrid mode sharding is a flexible data segmentation method that combines multiple sharding strategies to adapt to data of different types and characteristics. The specific implementation methods include: combining fixed-length and semantic block sharding, first performing preliminary segmentation according to semantic boundaries, and then using fixed-length supplements for parts that cannot be completely divided according to semantics; using differentiated sharding strategies for different data types, such as semantic block sharding for text, regional sharding for images, and key frame sharding for videos; dynamically adjusting sharding strategies, and adaptively selecting sharding methods based on data complexity and information entropy. For example, for highly structured data, sharding can be prioritized according to the internal logical structure; for unstructured or highly random data, equal-length sharding can be used.

[0035] The pre-trained sharded deep learning model construction and training process is as follows: The first task in building a sharded deep learning model is to collect and organize large-scale and diverse training datasets. The datasets should cover a variety of data forms such as text, images, videos, tables, etc., and include files of different types, sizes, and complexities.

[0036] Before model training, the original data needs to be systematically feature engineered. This process includes data standardization, dimensionality reduction, feature selection, and feature encoding. Different feature extraction methods are used for different types of data, such as word vectors for text data and convolutional features for image data. Through dimensionality reduction techniques such as principal component analysis (PCA), high-dimensional features are mapped to low-dimensional space, effectively reducing noise and extracting key information. Accurate feature engineering is a key link in improving model performance.

[0037] This model adopts the Multilayer Perceptron (MLP) architecture, which consists of an input layer, multiple hidden layers, and an output layer. The input layer receives a multidimensional feature vector, including data type encoding, sensitivity level, and information entropy. The hidden layer uses a fully connected layer with a ReLU activation function to enhance the nonlinear expression ability of the network. The output layer uses a linear regression algorithm to predict the shard size, number of shards, and boundary conditions, and realizes intelligent adjustment of the data sharding strategy.

[0038] The model training process uses standard deep learning optimization methods. Using Mean Squared Error (MSE) as the loss function, select optimization algorithms such as Adam (Adaptive Moment Estimation) or RMSprop (Root Mean Square Propagation) to gradually adjust the model parameters. Introduce random dropout and L2 regularization techniques to effectively prevent overfitting and improve the generalization ability of the model. Evaluate model performance through cross-validation methods, perform multiple rounds of iterations and hyperparameter tuning. The training process focuses on balancing the learning ability and generalization ability of the model to ensure good sharding performance on different data sets.

[0039] The multilayer perceptron is a feedforward neural network that works based on hierarchical feature extraction and nonlinear transformation. The input layer receives the original feature vector, and each input neuron corresponds to a feature dimension. The hidden layer is fully connected, with each neuron connected to all neurons in the previous layer, and linearly transformed using weight matrices and bias terms. Activation functions (such as ReLU) introduce nonlinearity, allowing the network to learn complex feature mappings.

[0040] Each layer is calculated by f(Wx + b), where W is the weight matrix, b is the bias term, and f is the activation function. The output layer uses linear output or activation functions such as softmax according to the task type (regression or classification) to map the learned features to the target space. Through the back propagation algorithm and gradient descent, the network weights are continuously adjusted to gradually approach the optimal parameters.

[0041] S13: Generate a pre-encrypted shard identifier for each data shard, wherein the pre-encrypted shard identifier includes: a global identifier of the original data, a shard sequence number, and a characteristic fingerprint of the data shard.

[0042] Specifically, the length of the global identifier is controlled within 8-16 characters, and supports user customization, which can use a combination of letters and numbers. The shard sequence number uses an incremental encoding method to clearly record the total number of shards and the current shard position. The format can be "current shard / total number of shards". Feature fingerprint extraction uses a fast hash algorithm, such as CRC32 (CRC32 is a cyclic redundancy check code used to detect errors in data transmission or storage, which can ensure that the data originally written to the image file is consistent with the data used from the image file), to quickly calculate the checksum of the data block. Finally, the system assembles the global identifier, shard sequence number, and feature fingerprint into a unique identifier, which is generated by string concatenation. This identifier generation method not only ensures the uniqueness and immutability of the data, but also reduces the computational complexity.

[0043] This multi-level, intelligent data segmentation and identification generation method makes full use of machine learning and information security technologies, and can preserve the integrity of the original data to the greatest extent while ensuring data security.

[0044] like Figure 3 As shown, step S20 specifically includes the following steps: S21: Perform security and sensitivity analysis on each data shard to obtain an encryption strength level.

[0045] Specifically, a multi-dimensional security assessment is conducted on each data shard, including data content sensitivity assessment, information entropy calculation, potential leakage risk analysis, and data structure complexity assessment. The multi-dimensional security assessment uses quantitative indicators, including sensitivity level (0-10), information entropy (0-8), and leakage risk coefficient (0-1), through which the security characteristics of the data shard are comprehensively measured.

[0046] The encryption strength level is determined based on the security analysis results. In this embodiment, the encryption strength is divided into five levels: low level (Level 1), medium-low level (Level 2), medium level (Level 3), high level (Level 4) and very high level (Level 5). Each level corresponds to a different sensitivity and information entropy range.

[0047] S22: Based on the encryption strength level, select and determine an adaptive adaptive encryption algorithm, where the adaptive encryption algorithm includes: a symmetric encryption algorithm, an asymmetric encryption algorithm, a hybrid encryption algorithm or a homomorphic encryption algorithm.

[0048] Specifically, the most suitable encryption algorithm is dynamically selected according to different encryption strength levels. For example, Level 1 uses AES-128 symmetric encryption, Level 2 uses AES-192, Level 3 uses RSA (Rivest-Shamir-Adleman, referred to as RSA, an asymmetric encryption algorithm) and AES (Advanced Encryption Standard, referred to as AES, a symmetric encryption algorithm) hybrid encryption, Level 4 uses Paillier homomorphic encryption, and Level 5 uses the more advanced BGV (Brakerski-Gentry-Vaikuntanathan) homomorphic encryption scheme. This dynamic and adaptive algorithm selection mechanism can provide the optimal encryption strategy based on the actual security needs of the data.

[0049] S23: Generate a unique key credential for each data slice according to the adaptive encryption algorithm, wherein the key credential includes: a slice identifier before encryption, an encryption key, and a key length.

[0050] Specifically, for each data shard, the system generates a unique key credential that contains the shard identifier before encryption. The key generation process takes encryption strength into account and dynamically generates encryption keys that match the encryption algorithm and security level. The length and complexity of the key are directly related to the sensitivity of the data shard, ensuring reliable encryption protection even in high-risk scenarios.

[0051] S24: Perform security verification on the key certificate.

[0052] Specifically, after the key certificate is generated, the system will perform multiple security verifications, including key randomness detection, key collision check, encryption algorithm compatibility verification, and key length security assessment. These verification processes can effectively eliminate potential security risks and ensure that each key certificate has undergone security review.

[0053] S25: Store the key certificate that has passed security verification in the key management system.

[0054] Specifically, key credentials that have passed security verification will be stored in a dedicated key management system, which has real-time monitoring and dynamic update capabilities, and can continuously track and manage the life cycle of keys.

[0055] The data sharding adaptive encryption method proposed in this embodiment builds a comprehensive, flexible and efficient data security storage solution through multi-dimensional security analysis, dynamic encryption algorithm selection, refined key generation and strict security verification.

[0056] like Figure 4 As shown, step S30 specifically includes the following steps: Step S31: Determine the type of homomorphic encryption algorithm, where the type of homomorphic encryption algorithm includes a partial homomorphic encryption algorithm or a fully homomorphic encryption algorithm.

[0057] Specifically, the choice of homomorphic encryption algorithm type is based on the security requirements and computational complexity of the data sharding. Partially homomorphic encryption algorithms, such as the Paillier algorithm, only support specific types of homomorphic operations, such as addition or multiplication, and are suitable for scenarios with relatively low computational complexity requirements. Fully homomorphic encryption algorithms, such as the BGV (Brakerski-Gentry-Vaikuntanathan) scheme, can perform arbitrarily complex computational operations on encrypted data, providing more comprehensive encryption protection. The choice of algorithm type needs to comprehensively consider data sensitivity, computational overhead, and performance requirements. For highly sensitive data shards that require complex calculations, it is recommended to use a fully homomorphic encryption algorithm.

[0058] Step S32: Obtain the public key preset in the key management system.

[0059] Specifically, these public keys are pre-generated and distributed by a dedicated key generation center to ensure the security and reliability of the public keys. The selection of public keys takes into account multiple factors, including key length, algorithm complexity, and security level. For different homomorphic encryption algorithms, the generation and management of public keys are also different. For example, the public key of the Paillier algorithm is generated by two large prime numbers, while the public key of the BGV scheme involves more complex parameter settings. In practical applications, the system will regularly update the public key to deal with potential cryptanalysis attacks.

[0060] Step S33: Convert the key certificate of each data shard into a standard data format compatible with the homomorphic encryption algorithm to obtain a standardized key certificate.

[0061] Specifically, for the key credential of each data shard, the system needs to perform a data conversion process. This process includes the reorganization of the data structure, the adjustment of the encoding method, and the normalization of metadata. The purpose of standardization is to eliminate the differences between different data formats and ensure that the key credential can be seamlessly integrated into the homomorphic encryption process. Specifically, the conversion process includes the following operations: converting the key information of the original key credential (such as the shard identifier before encryption, encryption key, and key length) into a specific data structure, adjusting the data encoding method to match the requirements of the target homomorphic encryption algorithm, and ensuring that the converted data format can retain all the key information of the original key credential.

[0062] Step S34: Use the public key to homomorphically encrypt the standardized key certificate to generate a corresponding encryption handle.

[0063] Specifically, the system uses the public key obtained in step S32 to homomorphically encrypt the standardized key credential and generate a corresponding encryption handle. Homomorphic encryption allows data to be calculated in an encrypted state without decryption. For key credentials, the system can still securely locate, match and manage keys even in an encrypted state. The generation process of the encryption handle follows strict encryption algorithm specifications. For example, in the Paillier algorithm, the generation of the encryption handle is based on large prime numbers and random numbers in the public key; in the BGV scheme, the generation of the encryption handle involves more complex polynomial rings and homomorphic operations. The encryption process ensures that the generated encryption handle is highly random and unpredictable, thereby providing security protection.

[0064] Step S35: Establish a mapping relationship between the encryption handle and the corresponding key certificate, and record the mapping relationship in the key management system.

[0065] Specifically, the system establishes a unique mapping relationship between each encryption handle and its corresponding original key credential, and records this mapping relationship in the key management system. This mapping mechanism provides a secure way to associate encryption handles and original key credential, so that the corresponding key credential can be quickly located and retrieved through the encryption handle when needed. The mapping relationship is recorded using multiple encryption and access control mechanisms to ensure that only authorized entities can access this sensitive information.

[0066] like Figure 5 As shown, step S40 specifically includes the following steps: Step S41: extracting the pre-encryption slice identifier, the encryption key and the key length from the key certificate corresponding to the current data slice to be encrypted.

[0067] In this step, the extraction process first locates the key credential associated with the current data shard to be encrypted. The key credential is stored in a secure key management system and contains multiple key information elements. The pre-encryption shard identifier is key information that uniquely identifies the data shard and is used to accurately locate and track the data shard before and after encryption. The encryption key is the key to perform encryption operations, and its security directly affects the strength of data protection. The key length reflects the complexity of the encryption algorithm, and the longer it is, the higher the security.

[0068] Step S42: Determine the fragment sequence number of the data fragment to be encrypted according to the fragment identifier before encryption.

[0069] In this step, by analyzing the pre-encryption shard identifier, the system can accurately locate the position of the current data shard in the original data set. The shard sequence number not only reflects the order of the data shards, but also provides an important basis for the subsequent generation of the post-encryption shard identifier. In practical applications, the pre-encryption shard identifier can be encoded in a variety of ways, such as a composite identifier containing a timestamp, data source information, and a sequential number.

[0070] Step S43: encrypting the data slice to be encrypted according to the encryption algorithm, the encryption key and the key length corresponding to the data slice to be encrypted, and generating an encrypted data slice; In this step, based on the extracted encryption key and key length, the system selects an appropriate encryption algorithm to encrypt the data shards. The encryption process must not only ensure the confidentiality of the data, but also maintain the integrity and availability of the data. Encryption algorithms include symmetric encryption (such as AES) and asymmetric encryption (such as RSA). The encryption process needs to consider multiple factors, such as the computational complexity of the encryption algorithm, encryption speed, and security level. Different encryption strategies are required for different types of data shards. For example, for highly sensitive personal information, more complex encryption algorithms and longer key lengths are required. The encryption process can also include necessary padding and mode selection to enhance the security of encryption.

[0071] Step S44: Based on the fragment sequence number, generate a corresponding encrypted fragment identifier for the encrypted data fragment.

[0072] In this step, based on the shard sequence number determined in step S42, the system needs to generate a unique and consistent encrypted shard identifier. This process not only retains the sequence number information of the original shard, but also incorporates the characteristics of the encryption process. The generation algorithm may include: using the hash value of the encrypted data, combining the encryption timestamp, adding the encryption algorithm identifier, etc.

[0073] Step S45: Execute steps S41 to S44 in a loop until all the data slices to be encrypted have completed the encryption process.

[0074] In this step, the system performs encryption processing on the data slices to be encrypted one by one through a loop iteration. To improve efficiency, you can consider using parallel encryption technology to support the simultaneous encryption of multiple data slices. The judgment standard for loop termination is that all the predetermined data slices have completed the encryption processing and generated the corresponding encrypted slice identifiers.

[0075] This method achieves precise positioning and personalized encryption of data shards by extracting the pre-encryption shard identifier, encryption key, and key length from the key credential, ensuring that each data shard can obtain encryption protection that matches its sensitivity. The mechanism of generating encrypted shard identifiers based on shard sequence numbers not only retains the order information of the original data shards, but also provides reliable technical support for subsequent data tracking and recovery. This method effectively prevents data leakage risks and improves the overall data security protection level through strict encryption key management and multi-level encryption strategies.

[0076] like Figure 6 As shown, step S50 specifically includes the following steps: Step S51: Obtain a first feature data set of each encrypted data shard and a second feature data set of multiple distributed storage nodes.

[0077] Specifically, the first feature data set reflects the inherent properties of encrypted data shards. The shard size determines the storage resource requirements, the encryption algorithm type affects the security requirements, and the sensitivity level determines the security level selection of the storage node. The second feature data set comprehensively describes the physical and performance characteristics of the distributed storage nodes. The geographic location coordinates take into account the geographical distribution of data and network latency, the storage capacity reflects the node storage capacity, the network bandwidth affects the data transmission efficiency, the load balancing status ensures the overall performance of the system, and the security level ensures the security of data storage.

[0078] Step S52: Determine the matching score between each encrypted data shard and each storage node based on the first feature data set, the second feature data set and the multi-dimensional distributed storage rules, and determine the target storage node corresponding to each encrypted data shard from multiple distributed storage nodes based on the multiple matching scores.

[0079] Specifically, the process considers the data shard characteristics and storage node attributes by constructing a multi-dimensional scoring model. The scoring model not only focuses on a single dimension, but also comprehensively evaluates various indicators to ensure the selection of the optimal storage node. The matching algorithm needs to balance multiple factors, such as storage cost, security, transmission efficiency, etc. Through sophisticated weight configuration and calculation model, personalized storage node matching can be achieved for each data shard, maximizing overall system performance and data security.

[0080] Step S53: Transmit and store the encrypted data segments to corresponding target storage nodes respectively.

[0081] Specifically, the transmission process needs to ensure data integrity, security, and efficiency. The system will use encrypted channels for transmission and perform data integrity checks during the transmission process. The transmission strategy needs to consider factors such as network bandwidth, node load, and geographical distribution. Optimization technologies such as parallel transmission or segmented transmission can be used to improve overall transmission efficiency.

[0082] Step S54: Record the location information of each encrypted data slice on the corresponding target storage node.

[0083] Specifically, the recording process is not just about simply storing the node address, but also about capturing the detailed storage context of the data shards. This includes storage timestamps, unique identifiers for data shards, storage node details, encryption methods, etc. The metadata management system needs to design a highly available and highly consistent storage and retrieval mechanism to ensure that data can be quickly located and restored even in the event of partial node failure.

[0084] This embodiment gradually calculates the matching degree of each dimension, comprehensively weighs different indicators, and finally selects the optimal storage node based on the preset threshold. The core of the algorithm is to establish a dynamic scoring model that can make the optimal storage decision based on the real-time system status and data characteristics.

[0085] like Figure 7 As shown, step S52 specifically includes the following steps: Step S521: configure five preset weights and the number N of multiple distributed storage nodes.

[0086] In this embodiment, the five preset weights correspond to the five key dimensions of geographical location (W1), storage capacity (W2), network bandwidth (W3), load balancing (W4) and basic security (W5). The weight configuration follows the principle of the sum of 1, that is, W1+W2+W3+W4+W5=1. The specific weights can be adjusted dynamically according to the actual business scenario. For example, for highly sensitive data such as finance and medical care, the basic security weight (W5) can be increased to 0.4 to 0.5; for large data volume transmission scenarios, the weights of network bandwidth (W3) and storage capacity (W2) can be relatively higher. The choice of the number of nodes N is the balance point between computational complexity and matching accuracy, and is selected in the range of 5-50 according to the system scale and performance requirements.

[0087] Step S522: According to the first feature data set of the current encrypted data shard and the second feature data set of the current distributed storage node, the geographic location matching degree, storage capacity matching degree, network bandwidth matching degree, load balancing matching degree and basic security matching degree are determined respectively.

[0088] In this embodiment, for the geographic location matching, the distance between the data shard source (or the location of the computer device that executes the data sharding) and the geographic coordinates of the storage node can be calculated, and a standardized distance decay function can be used. The closer the distance, the higher the matching degree. The storage capacity matching degree focuses on the degree of adaptation of the node's remaining storage space to the size of the data shard, and is comprehensively evaluated using capacity utilization and remaining space ratio. The network bandwidth matching degree is based on the node's real-time network throughput and data shard transmission requirements, and is evaluated through bandwidth utilization and transmission speed. The load balancing matching degree considers the node's current CPU, memory utilization, and the amount of stored data to ensure that the node is not over-stressed. The basic security matching degree takes the storage node as the object, and constructs a differentiated storage strategy by comprehensively analyzing the node's security level, access control capabilities, and historical security events. The basic security matching degree analysis includes three dimensions: first, the node security level is divided into three levels: L1 (0-3 points), L2 (4-7 points), and L3 (8-10 points) from the aspects of hardware configuration, system patch, firewall and intrusion detection; second, the access control capability is analyzed from the perspective of identity authentication strength, multi-factor authentication and permission control, and is divided into three levels: basic (0-3 points), medium (4-7 points) and strict (8-10 points); finally, according to the number and severity of security incidents in the past 12 months, historical security incidents are scored as no incident (8-10 points), minor incidents (4-7 points) and severe incidents (0-3 points). Finally, the formula: M_security = (security level score 0.4) + (Access Control Score 0.3) + (Historical Security Incident Score 0.3), calculate the comprehensive security matching degree, provide accurate storage node selection for data of different sensitivities, and ensure that highly sensitive data is stored in the storage node with the highest security.

[0089] Specifically, the formula for geographic location matching is as follows: Mg = 1 - (d / D). The geographic location matching formula quantifies the degree of geographic location matching by calculating the physical geographic distance between the data source location and the target storage node and normalizing the distance to the interval [0,1]. The parameter Mg represents the geographic location matching, and its value range is between 0 and 1. When the matching degree is close to 1, it means that the geographic locations are very close; when the matching degree is close to 0, it means that the geographic locations are very different. The parameter d represents the actual physical distance between the data shard source location and the storage node, calculated based on the geographic coordinate system, in kilometers (km). The parameter D represents the preset maximum distance threshold. When the actual distance exceeds this threshold, the matching degree will be considered the lowest.

[0090] Specifically, the storage capacity matching formula is as follows: Ms = min(1, S / R) × (1 - U). The storage capacity matching formula is used to evaluate whether the storage node has sufficient storage space and maintains a good resource utilization state. By comparing the available storage space and the data sharding requirements, and combining the current storage utilization, the formula provides an accurate storage node selection mechanism for the distributed storage system. The parameter Ms represents the storage capacity matching degree, and its value range is between 0 and 1. The closer the value is to 1, the more suitable the storage node is for storing new data shards. The parameter S represents the unallocated storage space currently available to the storage node, reflecting the actual space size that the node can currently use to store new data. The parameter R represents the actual storage space requirement of the data shard to be stored, describing the storage space size required for a specific data shard. The parameter U represents the storage utilization of the node, that is, the proportion of used storage space to the total storage space, and its value range is between [0, 1].

[0091] Specifically, the network bandwidth matching formula is as follows: Mb = min(1, B / W) × (1 - Ub). The network bandwidth matching formula is used to comprehensively evaluate the network transmission capacity and resource availability of storage nodes. By comparing the actual available bandwidth with the bandwidth required by the task and considering the current network bandwidth utilization, the formula provides a dynamic network resource matching strategy for the distributed storage system. The parameter Mb represents the network bandwidth matching degree, and its value range is between 0 and 1. The closer the value is to 1, the more suitable the node's network bandwidth is for the data transmission task. The parameter B represents the actual network transmission bandwidth currently available to the storage node, which reflects the node's network communication capability, usually in megabits per second (Mbps). The parameter W represents the minimum network bandwidth required by the data transmission task, and describes the minimum bandwidth requirement required to complete a specific data transmission task. The parameter Ub represents the actual usage ratio of the current network bandwidth, and its value range is between [0, 1]. The higher the value, the more fully the network resources are used.

[0092] Specifically, the load balancing matching degree formula is as follows: Ml = 1 - (0.4Uc + 0.3Um + 0.3Us). By weighted calculation of CPU utilization, memory utilization and storage load, this formula can fully reflect the overall load status of the node. Parameter Ml represents the load balancing matching degree, and its value range is between 0 and 1. The closer the value is to 1, the more ideal the load status of the node. Parameter Uc represents the resource occupancy ratio of the central processing unit, reflecting the computing resource usage of the node, and its value range is between [0, 1]. Parameter Um represents the resource usage ratio of the memory, describing the usage status of the node's memory resources, and its value range is between [0, 1]. Parameter Us represents the resource occupancy ratio of the storage system, reflecting the load status of the storage device, and its value range is [0, 1].

[0093] Specifically, the basic security matching degree formula is as follows: Mse = (Rs × 0.4) + (Ra × 0.3) + (Rh× 0.3). The basic security matching degree formula is used to comprehensively evaluate the security and trustworthiness of storage nodes. By comprehensively considering the security level, access control mechanism and historical security events of the node, the formula provides a multi-dimensional security evaluation indicator to help the distributed storage system select the safest and most reliable storage node. The parameter Mse represents the basic security matching degree, and its value range is between 0 and 1. The closer the value is to 1, the higher the security of the node. The parameter Rs represents the overall security level score of the node, which reflects the comprehensive evaluation of the node's security configuration, protection measures and security policies. The parameter Ra represents the effectiveness score of the node's access control mechanism, which evaluates the node's capabilities in identity authentication, permission management and access control. The parameter Rh represents the comprehensive score of the node's historical security events, which is based on the security events, vulnerabilities and attacks that have occurred in the node's history.

[0094] Step S523: Determine a comprehensive matching score according to the five preset weights, the geographical location matching degree, the storage capacity matching degree, the network bandwidth matching degree, the load balancing matching degree and the basic security matching degree.

[0095] In this embodiment, the comprehensive matching score is calculated using a weighted sum model, and the specific formula is: Comprehensive matching score = W1 Geographical Matching + W2 Storage capacity matching + W3 Network bandwidth matching + W4 Load balancing matching degree + W5 Basic security matching. The matching degree of each dimension is standardized to ensure that it is within the range of 0-1, so that different dimensions can be compared fairly. The weight configuration allows dynamic adjustment according to specific business scenarios, providing personalized storage node matching strategies for different types of data shards.

[0096] Step S524: loop through steps S522 and S523 to obtain N comprehensive matching scores.

[0097] In this embodiment, for N candidate storage nodes, the system will repeatedly perform the matching degree calculation and comprehensive scoring steps. Each node will undergo a rigorous multi-dimensional evaluation to ensure the fairness and comprehensiveness of the selection process.

[0098] Step S525: Determine the target storage node corresponding to the current encrypted data shard according to the N comprehensive matching scores and a preset score threshold.

[0099] In this embodiment, the target storage node is the most matching node corresponding to the current encrypted data shard, and the target storage node selection is based on comprehensive matching score sorting and threshold filtering. The preset threshold is used to filter nodes that meet the minimum storage quality requirements. The selection strategy is not only to select the node with the highest score, but also to consider the diversity of nodes and the overall load balance of the system. If the highest-scoring node is close to saturation, the system will select the suboptimal but lower-load node to achieve global performance optimization.

[0100] S526: Determine whether all encrypted data shards have determined the target storage node; S527: If the judgment result is no, step S522 to step S526 are executed in a loop until all encrypted data slices have determined the target storage node.

[0101] By checking whether all encrypted data shards have been matched with the best storage nodes, the system can continue to execute the matching algorithm until all encrypted data shards are properly stored. This mechanism provides a dynamic and adaptive storage node allocation framework that can cope with complex and changing storage environments and ensure the efficiency, security and reliability of data storage.

[0102] like Figure 8 As shown, step S60 specifically includes the following steps: Step S61: Generate a mapping table between the global identifier of the original data and the metadata information of each data slice.

[0103] Specifically, global identifier generation is the basis of metadata management. Global identifiers are generated using UUID (Universally Unique Identifier) ​​or content-based hashing algorithms to ensure that each original data set has a unique and non-repeatable identity tag. The mapping table records the logical relationship between the original data and its shards. When generating the mapping table, the system will catalog each data shard in detail and establish a many-to-many association mechanism from the original data to the shards.

[0104] Step S62: In the mapping table, the metadata information of each data shard is recorded according to predetermined standard data fields, and the standard data fields include: a global identification field of the original data, a shard identification field before encryption, a shard identification field after encryption, a location information field, an encryption algorithm field, and an encryption handle field of the key certificate.

[0105] Specifically, the design of standard data fields follows strict information management specifications, and each field carries specific semantics and functions. The global identification field of the original data serves as the primary index, providing a basic entry point for data tracing. The shard identification field before encryption records the original characteristics of the data shard, and the shard identification field after encryption reflects the unique mark after encryption conversion. The location information field records the storage location of each data shard in detail, including node address, storage path, etc. The encryption algorithm field identifies the specific algorithm type and version used to encrypt the shard, providing necessary information for data decryption.

[0106] Step S63: storing the mapping table in the distributed database of the metadata management server according to the database specification.

[0107] Specifically, the storage process follows strict database specifications and adopts a multi-copy, multi-node storage strategy to ensure high reliability and instant accessibility of metadata. The distributed storage architecture uses non-relational database technology similar to HBase. The storage solution takes into account complex engineering issues such as data partitioning, load balancing, and fault recovery. The metadata management server is not just a passive storage node, but also has advanced features such as intelligent routing, cache optimization, and real-time synchronization.

[0108] like Fig. 9 As shown, the method further comprises the following steps: Step S70: receiving a data reading request, wherein the data reading request carries the global identifier of the data to be read, the identity certificate of the requesting party, and access permission information.

[0109] Specifically, the global identifier is used as the unique locator of the data, the identity credentials of the requester are used for identity authentication, and the access rights information defines the specific scope and restrictions of data access. The global identifier uses a unique tag generated by a UUID or content hash algorithm to ensure accurate location of the data. The identity credentials can be in the form of digital certificates, tokens, or biometrics, and the access rights information describes in detail the specific operations that the requester can perform, such as read, partial read, or conditional read.

[0110] Step S80: Verify whether the data read request is legal based on the requester identity certificate and the access permission information.

[0111] Specifically, the verification process adopts a multi-layer security check strategy. First, the authenticity of the identity is confirmed through the encrypted signature and validity period of the identity credential, and then the access rights of the requester are accurately matched based on the Fine-Grained Access Control (FBAC) model. Permission verification not only checks whether the user has read permission, but also evaluates the specific scope, time, frequency, etc. of the reading. For highly sensitive data, the system can enable multi-factor authentication, such as combining dynamic tokens, geographic location or behavioral characteristics, to build a more stringent security verification system.

[0112] Step S90: When the verification result is legal, metadata information of multiple data shards is retrieved from the global mapping index according to the global identifier of the data to be read.

[0113] Specifically, the system uses global identifiers to quickly locate relevant data shards in the distributed metadata management system. The retrieval process not only obtains the physical storage location of the shard, but also extracts key metadata including encryption algorithms, shard sequence numbers, etc. For large-scale data, distributed indexing techniques such as inverted indexes or multidimensional indexes can be used to further optimize retrieval performance.

[0114] Step S100: Obtain the corresponding key certificate from the key management system according to the encryption handle of the key certificate in the metadata information of each data shard.

[0115] Specifically, through the encryption handle in the metadata, the system accurately locates and extracts the corresponding key credentials. The key management system uses a high-security key storage and retrieval mechanism, such as a hardware security module (HSM) or a cloud-native key management service. The acquisition process adds an additional layer of authentication and access control. Even if the previous verification is successfully passed, it still needs to be confirmed by the key management system for the second time. For data of different sensitivity levels, the system can adopt a hierarchical key management strategy to provide differentiated key protection strengths for different types of data.

[0116] Step S110: Determine the corresponding decryption algorithm according to the encryption algorithm in the metadata information of each of the data slices, and determine the decryption key according to the encryption key and key length in the key certificate.

[0117] Specifically, first extract the encryption algorithm type (such as AES-256-CBC) from the metadata and select the corresponding decryption algorithm; then, based on the key length (such as 256 bits) and key type (symmetric / asymmetric) recorded in the key credential, extract the matching decryption key from the key management system. For example, for data shards encrypted using AES-256-CBC, the system will call the AES decryption algorithm and use a 256-bit symmetric key for decryption. To ensure security, the system will also perform key validity period verification, integrity check, and uniqueness verification to prevent potential security risks. In complex scenarios with multiple shards and multiple algorithms, the system can intelligently identify the specific encryption method of each shard, and accordingly select and apply the correct decryption algorithm and key to ensure accurate data restoration and information security.

[0118] For symmetric encryption (such as AES), the same key is used for encryption and decryption. During the processing, the encryption algorithm type (such as AES-256-CBC) is first extracted from the metadata information, and then the corresponding decryption algorithm is selected based on the encryption algorithm type. Next, the encryption key and key length (such as 256 bits) are extracted from the key credential. When decrypting, the system uses the same key and decryption algorithm (such as the AES decryption algorithm) to restore the data. For example, in the case of AES-256-CBC encryption, a 256-bit symmetric key is used to decrypt the encrypted data. During the decryption process, key validity period verification, integrity check, and uniqueness verification are also required to ensure that the data has not been tampered with. In this way, secure decryption and restoration of data in symmetric encryption can be ensured.

[0119] Asymmetric encryption uses public and private keys to encrypt and decrypt data, and the private key is required for decryption. First, the encryption algorithm type (such as RSA or elliptic curve cryptography ECC) is extracted from the metadata information, and then the corresponding decryption algorithm is selected based on the type. Next, the system queries the corresponding private key through the key management system based on the pre-encryption shard identifier, encryption algorithm type, and key length in the key certificate. The key certificate contains metadata related to the encrypted shard, and the private key is stored in the key management system. The system determines to extract the corresponding private key (such as RSA private key) from the key management system based on this information in the key certificate. For RSA-encrypted data, the system will call the RSA decryption algorithm and use the extracted RSA private key to decrypt the data shard. Unlike symmetric encryption, the decryption and encryption processes of asymmetric encryption use different keys, so the security of the private key needs to be ensured. During the decryption process, the system will also verify the validity of the private key to ensure that it has not been leaked or tampered with, thereby preventing potential security risks.

[0120] Hybrid encryption combines the advantages of symmetric encryption and asymmetric encryption. In this encryption method, an asymmetric encryption algorithm (such as RSA) is used to encrypt the symmetric encryption key (such as AES key), and then a symmetric encryption algorithm (such as AES) is used to encrypt the actual data. During the decryption process, the system first uses the private key to decrypt the symmetric key (for example, using the RSA private key to decrypt the AES key), and then uses the decrypted symmetric key to decrypt the data. In this way, hybrid encryption can not only provide higher security, but also optimize the performance of the encryption process, because symmetric encryption is more suitable for encrypting large amounts of data than asymmetric encryption.

[0121] Homomorphic encryption allows calculations to be performed on encrypted data and the calculated results are obtained when decrypted. Its characteristic is that encrypted data can be directly calculated, and the result after decryption is the calculated data. During the processing, the encryption algorithm type (such as homomorphic encryption) is first extracted from the metadata information, and then the corresponding decryption algorithm is selected. Homomorphic encryption involves a pair of keys (public key and private key). The private key is required for decryption. The system retrieves the correct private key from the key management system through the pre-encryption shard identification, encryption algorithm type and other information in the key credential. The system uses the private key to decrypt the encrypted data to restore the original data or calculation results.

[0122] Step S120: Use the decryption algorithm and decryption key of each encrypted data slice to perform decryption processing to obtain multiple decrypted data slices.

[0123] Specifically, each data shard uses a matching decryption algorithm and key independently to ensure that even if some shards are encrypted differently, they can be accurately restored. The decryption process can use further cryptographic operations, such as padding removal, integrity verification, etc. To improve performance, the system can use parallel decryption technology to process multiple data shards simultaneously in a multi-core processor or distributed computing environment.

[0124] Step S130: According to the pre-encryption fragment identifiers with fragment sequence numbers or the post-encryption fragment identifiers with fragment sequence numbers of the multiple data fragments, the multiple decrypted data fragments are reassembled to obtain the data to be read.

[0125] Specifically, the system sorts and splices the decrypted data fragments according to the fragment sequence number to ensure the integrity and consistency of the original data. The reorganization algorithm not only pays attention to the data order, but also performs redundancy checks and integrity checks to prevent data loss or tampering during the reorganization process. For large-scale data, streaming reorganization technology can be used to achieve decryption and reorganization at the same time, improving processing efficiency in large data volume scenarios.

[0126] Embodiment 2 like Fig.10As shown, this embodiment provides a distributed storage system for data security, which includes: A data sharding module, used for sharding the original data to be stored, generating a plurality of data shards, and determining a global identifier of the original data and a pre-encryption shard identifier of each of the data shards; An encryption algorithm selection module, used to determine a corresponding adaptive encryption algorithm and key certificate for each of the data slices; A key management module, used to perform homomorphic encryption on the key credential of each data shard to generate an encryption handle; A data slice encryption module, used to encrypt each data slice according to the encryption algorithm and key certificate corresponding to each data slice, obtain multiple encrypted data slices, and determine the encrypted slice identifier of each encrypted data slice; A distributed storage module, used to disperse and store the plurality of encrypted data shards to a plurality of distributed storage nodes according to a multi-dimensional distributed storage rule, and record the location information of each of the encrypted data shards on the corresponding distributed storage node; The metadata management module is used to store a global mapping index indicating the corresponding relationship between the global identifier of the original data and the metadata information of each data slice.

[0127] In some embodiments, the data sharding module specifically includes: A feature extraction submodule is used to extract features from the original data, analyze the extracted features, obtain the type code, sensitivity level and information entropy of the original data, and determine a feature vector according to the type code, the sensitivity level and the information entropy; A sharding deep learning submodule, for using a pre-trained sharding deep learning model, dynamically determining sharding parameters according to the feature vector, and sharding the original data according to the sharding parameters to generate a plurality of data shards; The identification generation submodule is used to generate a pre-encrypted segment identification for each data segment, wherein the pre-encrypted segment identification includes: a global identification of the original data, a segment sequence number, and a characteristic fingerprint of the data segment.

[0128] In some embodiments, the encryption algorithm selection module specifically includes: The security analysis submodule is used to perform security and sensitivity analysis on each data shard to obtain the encryption strength level; An algorithm selection submodule, used for selecting and determining an adaptive encryption algorithm based on the encryption strength level; A key generation submodule, used to generate a unique key credential for each of the data slices according to the adaptive encryption algorithm; A security verification submodule, used to perform security verification on the key certificate; The storage submodule is used to store the key credentials that have passed security verification in the key management system.

[0129] In some embodiments, the key management module specifically includes: Homomorphic encryption algorithm selection submodule, used to determine the type of homomorphic encryption algorithm; The public key acquisition submodule is used to obtain the public key preset in the key management system; A key credential conversion submodule, used to convert the key credential of each of the data shards into a standard data format compatible with the homomorphic encryption algorithm; An encryption processing submodule, used to use the public key to perform homomorphic encryption on the standardized key credential to generate a corresponding encryption handle; The mapping relationship recording submodule is used to establish a mapping relationship between the encryption handle and the corresponding key certificate, and record the mapping relationship in the key management system.

[0130] In some embodiments, the data sharding encryption module specifically includes: The key extraction submodule is used to extract the pre-encryption shard identifier, encryption key and key length from the key certificate corresponding to the current data shard to be encrypted; A fragment sequence number determination submodule, used to determine the fragment sequence number of the data fragment to be encrypted currently according to the fragment identifier before encryption; The encryption processing submodule is used to perform encryption processing on the current data slice to be encrypted according to the encryption algorithm corresponding to the current data slice to be encrypted, the encryption key and the key length, and generate an encrypted data slice; An identifier generation submodule, used to generate a corresponding encrypted fragment identifier for the encrypted data fragment based on the fragment sequence number; The cyclic encryption submodule is used to cyclically execute key extraction, shard sequence number determination, encryption processing and identifier generation until all data shards to be encrypted have completed encryption processing.

[0131] In some embodiments, the distributed storage module specifically includes: A feature data acquisition submodule, used to acquire a first feature data set of each encrypted data shard and a second feature data set of multiple distributed storage nodes; A matching score calculation submodule, used to determine a matching score between each encrypted data shard and each storage node according to the first feature data set, the second feature data set and a multi-dimensional distributed storage rule; A target storage node selection submodule, used to determine a target storage node corresponding to each of the encrypted data shards from a plurality of distributed storage nodes according to a plurality of the matching scores; A data transmission and storage submodule, used to transmit and store the encrypted data slices to corresponding target storage nodes respectively; The location information recording submodule is used to record the location information of each encrypted data slice on the corresponding target storage node.

[0132] In some embodiments, the matching score calculation submodule specifically includes: The weight configuration submodule is used to configure five preset weights and the number N of multiple distributed storage nodes; A matching degree calculation submodule, used to determine the geographical location matching degree, storage capacity matching degree, network bandwidth matching degree, load balancing matching degree and basic security matching degree respectively according to the first feature data set of the current encrypted data shard and the second feature data set of the current distributed storage node; A comprehensive matching score calculation submodule, used to determine a comprehensive matching score according to the five preset weights and each matching degree; A target node selection submodule, used to determine the most matching target storage node corresponding to the current encrypted data shard according to the N comprehensive matching scores and a preset score threshold; The loop processing submodule is used to loop through the matching degree calculation and the comprehensive matching score calculation until the best matching target storage node is determined for all encrypted data shards.

[0133] In some embodiments, the metadata management module specifically includes: A mapping table generation submodule is used to generate a mapping table of the global identifier of the original data and the metadata information of each data shard; A metadata recording submodule, used to record metadata information of each data slice in the mapping table according to a predetermined standard data field; The database storage submodule is used to store the mapping table in the distributed database of the metadata management server according to the database specification.

[0134] In some embodiments, the system further comprises: A data reading request receiving module, used to receive a data reading request, wherein the data reading request carries a global identifier of the data to be read, an identity certificate of the requesting party, and access permission information; A request legitimacy verification module, used to verify whether the data read request is legal based on the requester identity certificate and the access permission information; A metadata retrieval module, used for retrieving metadata information of multiple data shards in a global mapping index according to the global identifier of the data to be read when the verification result is legal; A key certificate acquisition module, used to obtain the corresponding key certificate from the key management system according to the encryption handle of the key certificate in the metadata information of each data shard; A decryption algorithm determination module, used to determine a corresponding decryption algorithm according to the encryption algorithm in the metadata information of each of the data slices, and determine a decryption key according to the encryption key and key length in the key certificate; The data decryption and reorganization module is used to perform decryption processing using the decryption algorithm and decryption key of each encrypted data slice to obtain multiple decrypted data slices, and reorganize the multiple decrypted data slices according to the slice identifiers of the multiple data slices to obtain the data to be read.

[0135] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0136] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, any of the above-mentioned distributed storage methods for data security is implemented.

[0137] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0138] The present invention also provides an electronic device. The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the distributed storage method for data security provided by the present invention.

[0139] Reference below Fig.11 , which shows a schematic diagram of the structure of a computer system 800 of an electronic device suitable for implementing an embodiment of the present invention. Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Fig.11 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the computer system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0141] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0142] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the main step diagram. In the above embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit 801, the above functions defined in the system of the present invention are executed.

[0143] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed storage method for data security, characterized in that: The following steps are involved: S10: Slice the original data to be stored to generate multiple data slices, and determine the global identifier of the original data and the pre-encryption slice identifier of each data slice having a slice sequence number; S20: for each of the data slices, respectively determine a corresponding adaptive encryption algorithm and key certificate, and store the key certificate of each of the data slices in a key management system; S30: performing homomorphic encryption processing on the key certificate of each data shard, and using the obtained encryption result as the encryption handle of the key certificate, wherein the encryption handle is used to locate and retrieve the corresponding key certificate in the key management system; S40: encrypting each of the data slices according to the encryption algorithm and key certificate corresponding to each of the data slices to obtain a plurality of encrypted data slices, and determining an encrypted slice identifier having a slice sequence number of each of the encrypted data slices; S50: According to the multi-dimensional distributed storage rule, the plurality of encrypted data shards are dispersedly stored in a plurality of distributed storage nodes, and the location information of each of the encrypted data shards on the corresponding distributed storage node is recorded; S60: storing a global mapping index indicating a correspondence between a global identifier of the original data and metadata information of each data slice in a metadata management server; The metadata information of each data shard includes: the pre-encryption shard identifier, the post-encryption shard identifier, the location information, the encryption algorithm, and the encryption handle of the key certificate corresponding to the data shard.

2. The method according to claim 1, characterized in that Step S10 specifically includes: S11: extracting features from the original data, and analyzing the extracted features to obtain the type code, sensitivity level and information entropy of the original data; determining a feature vector according to the type code, the sensitivity level and the information entropy; S12: using a pre-trained sharding deep learning model, dynamically determining sharding parameters according to the feature vector, and sharding the original data according to the sharding parameters to generate multiple data shards; S13: Generate a pre-encrypted shard identifier for each data shard, wherein the pre-encrypted shard identifier includes: a global identifier of the original data, a shard sequence number, and a characteristic fingerprint of the data shard.

3. The method according to claim 1, characterized in that Step S20 specifically includes: S21: Perform security and sensitivity analysis on each data shard to obtain an encryption strength level; S22: Based on the encryption strength level, select and determine an adaptive adaptive encryption algorithm, where the adaptive encryption algorithm includes: a symmetric encryption algorithm, an asymmetric encryption algorithm, a hybrid encryption algorithm, or a homomorphic encryption algorithm; S23: Generate a unique key credential for each data slice according to the adaptive encryption algorithm, wherein the key credential includes: a slice identifier before encryption, an encryption key, and a key length; S24: Perform security verification on the key certificate; S25: Store the key certificate that has passed security verification in the key management system.

4. The method according to claim 1, characterized in that: Step S30 specifically includes: S31: Determine the type of a homomorphic encryption algorithm, where the type of the homomorphic encryption algorithm includes a partially homomorphic encryption algorithm or a fully homomorphic encryption algorithm; S32: Obtain a public key preset in the key management system; S33: Convert the key certificate of each data shard into a standard data format compatible with the homomorphic encryption algorithm to obtain a standardized key certificate; S34: Using the public key to homomorphically encrypt the standardized key credential to generate a corresponding encryption handle; S35: Establish a mapping relationship between the encryption handle and the corresponding key certificate, and record the mapping relationship in the key management system.

5. The method according to claim 3, characterized in that: Step S40 specifically includes: S41: extracting the pre-encryption slice identifier, the encryption key and the key length from the key certificate corresponding to the current data slice to be encrypted; S42: Determine the fragment sequence number of the data fragment to be encrypted according to the fragment identifier before encryption; S43: Encrypt the data slice to be encrypted according to the encryption algorithm, the encryption key and the key length corresponding to the data slice to be encrypted, and generate an encrypted data slice; S44: Based on the fragment sequence number, generate a corresponding encrypted fragment identifier for the encrypted data fragment; S45: Execute steps S41 to S44 in a loop until all the data segments to be encrypted have completed the encryption process.

6. The method according to claim 1, characterized in that Step S50 specifically includes: S51: Obtain a first feature data set for each encrypted data shard and a second feature data set for multiple distributed storage nodes; the first feature data set includes: shard size, encryption algorithm type and sensitivity level; the second feature data set includes: geographic location coordinates of distributed storage nodes, storage capacity of distributed storage nodes, network bandwidth of distributed storage nodes, load balancing status of distributed storage nodes, and security level of distributed storage nodes; S52: Determine a matching score between each encrypted data shard and each storage node according to the first feature data set, the second feature data set, and the multi-dimensional distributed storage rule, and determine a target storage node corresponding to each encrypted data shard from multiple distributed storage nodes according to the multiple matching scores; S53: Transmitting and storing the encrypted data slices to corresponding target storage nodes respectively; S54: Record the location information of each encrypted data slice on the corresponding target storage node.

7. The method according to claim 6, characterized in that Step S52 specifically includes: S521: configuring five preset weights and the number N of multiple distributed storage nodes; S522: Determine the geographic location matching degree, storage capacity matching degree, network bandwidth matching degree, load balancing matching degree and basic security matching degree respectively according to the first feature data set of the current encrypted data shard and the second feature data set of the current distributed storage node; S523: Determine a comprehensive matching score according to the five preset weights, the geographical location matching degree, the storage capacity matching degree, the network bandwidth matching degree, the load balancing matching degree and the basic security matching degree; S524: loop through steps S522 and S523 to obtain N comprehensive matching scores; S525: Determine the target storage node corresponding to the current encrypted data shard according to the N comprehensive matching scores and a preset score threshold; S526: Determine whether all encrypted data shards have determined the target storage node; S527: If the judgment result is no, step S522 to step S526 are executed in a loop until all encrypted data slices have determined the target storage node.

8. The method according to claim 1, characterized in that Step S60 specifically includes: S61: Generate a mapping table of the global identifier of the original data and the metadata information of each data slice; S62: In the mapping table, metadata information of each data slice is recorded according to a predetermined standard data field, wherein the standard data field includes: a global identification field of original data, a slice identification field before encryption, a slice identification field after encryption, a location information field, an encryption algorithm field, and an encryption handle field of a key certificate; S63: According to database specifications, the mapping table is stored in a distributed database of the metadata management server.

9. The method according to claim 1, characterized in that: The method further comprises: S70: Receive a data read request, where the data read request carries a global identifier of the data to be read, an identity credential of the requesting party, and access permission information; S80: Verify whether the data read request is legal according to the requester identity certificate and the access permission information; S90: When the verification result is legal, metadata information of multiple data shards is retrieved from the global mapping index according to the global identifier of the data to be read; S100: Obtaining a corresponding key certificate from the key management system according to the encryption handle of the key certificate in the metadata information of each data shard; S110: Determine a corresponding decryption algorithm according to the encryption algorithm in the metadata information of each of the data slices, and determine a decryption key according to the encryption key and key length in the key certificate; S120: Perform decryption processing using the decryption algorithm and decryption key of each encrypted data slice to obtain multiple decrypted data slices; S130: According to the pre-encryption fragment identifiers with fragment sequence numbers or the post-encryption fragment identifiers with fragment sequence numbers of the multiple data fragments, the multiple decrypted data fragments are reassembled to obtain the data to be read.

10. A distributed storage system for data security, characterized in that: include: The data sharding module is used to perform sharding processing on the original data to be stored, generate multiple data shards, and determine the global identifier of the original data and the shard identifier before encryption of each data shard; An encryption algorithm selection module is used to adaptively determine the encryption algorithm and key credentials for each data shard, and store the key credentials in a key management system; A key encryption module, used to perform homomorphic encryption on the key credential to generate an encryption handle, wherein the encryption handle is used to locate and retrieve the corresponding key credential in the key management system; A data shard encryption module, used to encrypt the data shards according to the selected encryption algorithm and key certificate, generate encrypted data shards, and determine the encrypted shard identifier of each encrypted data shard; A distributed storage scheduling module, used to disperse and store the encrypted data shards to multiple distributed storage nodes according to multi-dimensional storage rules, and record the location information of each encrypted data shard; A metadata management server is used to store a global mapping index indicating the correspondence between the global identifier of the original data and the metadata information of each data shard, wherein the metadata information includes: the shard identifier of the data shard before encryption, the shard identifier after encryption, the location information, the encryption algorithm and the encryption handle of the key certificate.

Images