A method and system for storing identification encryption of distributed data element clusters

Abstract

The present invention discloses an identification encryption storage method and system for distributed data element clusters, which relates to the technical field of data encryption and storage, including: dividing a data set into data segments and generating a data segment set; integrating the features extracted from each data type into a comprehensive feature vector; selecting a clustering algorithm, and performing a clustering operation according to the selected clustering algorithm to divide the data into clusters with similar features, each cluster automatically constructing a data segment; assigning an identifier to each data segment and generating an identifier set; forming a check value set; generating an encrypted data segment set and a corresponding encryption key set; and distributing and storing the encrypted data segments. The identification encryption storage method for distributed data element clusters provided by the present invention optimizes the data storage and retrieval process through data segment processing and encryption strategies. By analyzing the sensitivity and communication frequency of data segments.

Description

Technical Field

[0001] The present invention relates to the field of data encryption and storage technology, and specifically to an identification encryption storage method and system for a distributed data element cluster. Background Art

[0002] As we enter the information age, traditional data storage and management methods face many challenges due to the rapid increase in data volume and the growing demand for distributed storage.

[0003] In the cloud computing and big data environment, how to ensure data security, integrity and efficient data retrieval has become a key technical challenge. Traditional methods usually rely on centralized data management systems, which not only increases the burden of data communication, but also increases the risk of data leakage. In addition, these systems often lack flexibility and have difficulty maintaining efficient data processing and retrieval speeds when dealing with highly dynamic data access patterns and complex network environments.

[0004] In distributed systems, data management is one of the core issues. In cloud storage and big data environments, how to effectively segment, encrypt and store data fragments while ensuring data security and fast access has become the focus of industry attention. Traditional data processing methods usually rely on simple encryption and fixed storage solutions, which often cannot meet the growing data security needs and processing efficiency.

[0005] Therefore, an identification encryption storage method for distributed data element clusters is needed to effectively balance security, storage cost and access efficiency when facing large-scale data processing. Summary of the invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is: the existing data storage methods have the problems of insufficient security and low storage and retrieval efficiency, especially when processing large-scale distributed data, how to optimize the data encryption, storage and retrieval process to improve security and efficiency.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for storing the identification encryption of a distributed data element cluster, comprising:

[0009] Divide the data set into data segments and generate a set of data segments;

[0010] Generating a set of data segments includes identifying data types in a data set and performing feature extraction; the data types include text data, numerical data, time series data, and image data;

[0011] Integrate the features extracted from each data type into a comprehensive feature vector;

[0012] Use principal component analysis to evaluate the main components of the feature vector, quantify the principal component analysis results, select a clustering algorithm, perform clustering operations based on the selected clustering algorithm, split the data into clusters with similar characteristics, and automatically construct each cluster into data segments;

[0013] Assign an identifier to each data fragment and generate an identifier set;

[0014] Use a hash algorithm to generate a check value for each data fragment to form a check value set;

[0015] Encrypt each data fragment to generate a set of encrypted data fragments and a corresponding encryption key set;

[0016] Distribute and store the encrypted data fragments;

[0017] When a data retrieval request is made by receiving an identifier of a data fragment, the corresponding encrypted data fragment is searched according to the identifier, the corresponding encryption key is obtained, the data fragment is decrypted, and the hash value of the decrypted data is compared with the check value generated before storage, and the original data is returned.

[0018] As a preferred solution of the identification encryption storage method of the distributed data element cluster described in the present invention, wherein: the feature extraction includes identifying the recognition language complexity, text length, key topics or emotional tendencies of the text data, and using NLP technology to extract semantic features and emotional analysis results;

[0019] Identify the statistical distribution range, standard deviation, kurtosis and skewness of numerical data, and calculate statistical descriptors, including mean, median, and variance;

[0020] Identify the periodicity, trend, and seasonal changes of time series data and extract Fourier transform features;

[0021] Identify the resolution, color depth, and content features of image data, and use convolutional neural networks to extract color histograms;

[0022] The selection clustering algorithm includes standardizing the comprehensive feature vector, calculating the covariance matrix between features based on the standardized features, and using the covariance matrix to identify the principal components that maximize the capture of the variation in the data using eigenvalue decomposition.

[0023] As a preferred solution of the identification encryption storage method of the distributed data element cluster described in the present invention, the standardization processing is expressed as:

[0024]

[0025] Among them, Z i represents the i-th feature after standardization, X i represents the i-th eigenvalue in the original data, μ i represents the mean of the i-th feature, σ i represents the standard deviation of the i-th feature, β i represents the adjustment coefficient dynamically calculated based on the coefficient of variation of feature i;

[0026] The covariance matrix is ​​expressed as,

[0027]

[0028] Where C represents the weighted covariance matrix, W represents the diagonal weight matrix, and Z k represents the standardized feature vector of the kth sample, n represents the number of samples, and T represents the transposition operator;

[0029] The eigenvalue decomposition is performed after the weighted covariance matrix C is obtained by using the QR algorithm.

[0030] As a preferred solution of the identification encryption storage method of the distributed data element clusters described in the present invention, wherein: the principal components that maximize the capture of variation in the data include selecting the K-means algorithm for clustering if the first three principal components explain more than 75% of the total variation and the components are dominated by numerical features;

[0031] If the numerical features are not dominant or the first three principal components fail to explain more than 75% of the data variation, analyze the relationship between the data features and use the scatter plot matrix and correlation coefficient matrix to evaluate the linear and nonlinear relationships between the data features and the overall distribution complexity of the features;

[0032] If the metric of the nonlinear relationship between data features exceeds the preset threshold, and the uniform distribution degree in the feature space exceeds the set consistency standard threshold, the spectral clustering algorithm is used;

[0033] If the point density is higher than two standard deviations of the average density of the overall data points, and the number of noise points in the data accounts for more than 5% of the total number of data points, the DBSCAN algorithm is selected;

[0034] If the data feature relationship analysis does not show significant nonlinear relationships or uneven density, and the number of noise points does not meet the criteria for using the DBSCAN algorithm, the hierarchical clustering algorithm is used.

[0035] As a preferred solution of the identification encryption storage method of the distributed data element cluster described in the present invention, wherein: the generating the identifier set includes performing attribute analysis on each data fragment, evaluating the sensitivity level in the network environment and the expected frequency of data communication;

[0036] Based on the analysis results, an identifier is generated that includes information about the network location and security level of the data fragment;

[0037] Bind identifiers to corresponding data fragments to optimize data routing, storage, and secure processing in the network.

[0038] As a preferred solution of the identification encryption storage method of the distributed data element cluster described in the present invention, wherein: the encrypting each data fragment includes analyzing the identifier of each data fragment, selecting an encryption algorithm according to the sensitivity level provided by the identifier; applying the selected encryption algorithm to encrypt the data fragment, each data fragment is encrypted independently;

[0039] Generate an independent encryption key for each encrypted data fragment, bind each encryption key to the corresponding encrypted data fragment, and integrate all encrypted data fragments and their corresponding encryption keys to form a set of encrypted data fragments.

[0040] As a preferred solution of the identification encryption storage method of the distributed data element cluster described in the present invention, wherein: the distributed storage of the encrypted data fragments includes performing attribute analysis on each encrypted data fragment, determining the data volume by analyzing the number of data points in the data fragment, and setting a storage strategy according to the sensitivity level contained in the identifier associated with the data fragment;

[0041] For data fragments whose volume exceeds a predetermined threshold, further sharding is performed to meet the needs of distributed storage;

[0042] A storage node is selected for each data segment based on the location information of the data segment and the expected frequency of data communication.

[0043] Another object of the present invention is to provide an identification encryption storage system for distributed data element clusters, which can solve the problems of insufficient security and low efficiency in traditional data storage methods by constructing an identification encryption storage system for distributed data element clusters, ensure the security of data during transmission and storage, and optimize data retrieval speed and storage efficiency.

[0044] To solve the above technical problems, the present invention provides the following technical solutions: an identification encryption storage system for a distributed data element cluster, comprising: a data segmentation module, a data identification module, a data verification module, a data encryption module, a data storage module and a data decryption module; the data segmentation module is used to segment a data set into data segments and generate a data segment set; the data identification module is used to assign an identifier to each data segment and generate an identifier set; the data verification module is used to generate a verification value for each data segment using a hash algorithm to form a verification value set; the data encryption module is used to encrypt each data segment and generate an encrypted data segment set and a corresponding encryption key set; the data storage module is used to store the encrypted data segments in a distributed manner;

[0045] The data decryption module is used to find the corresponding encrypted data segment according to the identifier when receiving a data segment for data retrieval request, obtain the corresponding encryption key, decrypt the data segment, and compare the hash value of the decrypted data with the check value generated before storage, and return the original data.

[0046] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the identification encryption storage method of the distributed data element cluster as described above when executing the computer program.

[0047] A computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the identification encryption storage method for a distributed data element cluster as described above.

[0048] Beneficial effects of the present invention: The identification encryption storage method of the distributed data element cluster provided by the present invention optimizes the data storage and retrieval process through data fragment processing and encryption strategy. By analyzing the sensitivity and communication frequency of the data fragment, identifiers and encryption strategies are assigned to the data fragment, thereby improving the efficiency of data routing, storage and security processing in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0050] Figure 1 An overall flow chart of an identification encryption storage method for a distributed data element cluster provided for one embodiment of the present invention.

[0051] Figure 2An overall structural diagram of an identification encryption storage system for a distributed data element cluster provided for the second embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0054] Example 1

[0055] Reference Figure 1 , as an embodiment of the present invention, provides an identification encryption storage method for a distributed data element cluster, comprising:

[0056] Divide the data set into data segments and generate a set of data segments;

[0057] Assign an identifier to each data fragment and generate an identifier set;

[0058] Use a hash algorithm to generate a check value for each data fragment to form a check value set;

[0059] Encrypt each data fragment to generate a set of encrypted data fragments and a corresponding encryption key set;

[0060] Distribute and store the encrypted data fragments;

[0061] When a data retrieval request is made by receiving an identifier of a data fragment, the corresponding encrypted data fragment is searched according to the identifier, the corresponding encryption key is obtained, the data fragment is decrypted, and the hash value of the decrypted data is compared with the check value generated before storage, and the original data is returned.

[0062] Traditional methods usually rely on static rules to segment data, based only on data size or simple time windows, without considering the actual content and type of the data. Traditional segmentation methods cannot distinguish different types of data in detail, resulting in inefficient processing, especially in environments with large-scale and multi-type data. Failure to optimize storage and processing strategies based on data type will lead to waste of resources, and using the same storage solution for text and image data will fail to realize the potential value of various types of data.

[0063] The present invention uses an algorithm to automatically identify data types and classify them according to the actual characteristics of the data. Such intelligent identification not only improves the accuracy of segmentation, but also provides a more suitable basis for subsequent data processing. By implementing different processing and storage strategies for different types of data, the efficiency and response speed of data processing are improved. Not only is the data type considered when segmenting data, but the storage and processing strategies are also dynamically adjusted according to the data access frequency and update frequency. This dynamic adjustment is difficult to achieve with traditional methods, and it makes data management more flexible and effective.

[0064] Generating a set of data segments includes identifying data types in a data set and performing feature extraction; the data types include text data, numerical data, time series data, and image data;

[0065] Identify the language complexity, text length, key topics or sentiment of text data, and use NLP technology to extract semantic features and sentiment analysis results;

[0066] Identify the statistical distribution range, standard deviation, kurtosis and skewness of numerical data, and calculate statistical descriptors, including mean, median, and variance;

[0067] Identify the periodicity, trend, and seasonal changes of time series data and extract Fourier transform features;

[0068] Identify the resolution, color depth, and content features of image data, and use convolutional neural networks to extract color histograms;

[0069] Integrate the features extracted from each data type into a comprehensive feature vector;

[0070] Use principal component analysis to evaluate the main components of the feature vector, quantify the principal component analysis results, select a clustering algorithm, and perform clustering operations based on the selected clustering algorithm to divide the data into clusters with similar characteristics. Each cluster is automatically constructed into data segments.

[0071] Principal component analysis PCA creates new variables that are linear combinations of the variables in the original data. The goal is to reduce the complexity of the entire dataset while retaining the most important features, i.e. those that explain most of the variation (difference) in the data.

[0072] The first three principal components were chosen because they explain most of the variation in the data, and if these components explain more than 75% of the variation, they are considered to capture the most important structure and patterns in the data set.

[0073] Focusing on the first three principal components makes data processing simpler and more efficient, especially when decision clustering methods are needed. This not only reduces the computational burden but also maintains the accuracy of data analysis.

[0074] In statistics, variation refers to how dispersed the data points are around the mean. In PCA, variation also refers to how dispersed the data are on each principal component. The goal of principal component analysis is to find those components that maximize the explanation of the variation in the data.

[0075] The selection clustering algorithm includes standardizing the comprehensive feature vector, calculating the covariance matrix between features based on the standardized features, and using the covariance matrix to identify the principal components that maximize the capture of the variation in the data using eigenvalue decomposition.

[0076] The standardization process is expressed as,

[0077]

[0078] Among them, Z i represents the i-th feature after standardization, X i represents the i-th eigenvalue in the original data, μ i represents the mean of the i-th feature, σ i represents the standard deviation of the i-th feature, β i represents the adjustment coefficient dynamically calculated based on the coefficient of variation of feature i;

[0079] The coefficient of variation is a statistical measure of relative dispersion, defined as the ratio of the characteristic standard deviation to the mean, expressed as,

[0080]

[0081] Among them, σ i is the standard deviation of feature i, μ i is the mean of feature i.

[0082] Adjustment coefficient β i Based on the coefficient of variation CV i To ensure a fairer comparison between different features, use the inverse of the coefficient of variation or a function of it to define β i , expressed as,

[0083]

[0084] The definition method makes the features with higher coefficient of variation (i.e. features with large relative dispersion) have lower weights in the standardization process, and vice versa.

[0085] The covariance matrix is ​​expressed as,

[0086]

[0087] Where C represents the weighted covariance matrix, W represents the diagonal weight matrix, and Z k represents the standardized feature vector of the kth sample, n represents the number of samples, and T represents the transposition operator;

[0088] Dynamically adjust the weight w of each feature based on real-time data or performance feedback ii The adjusted weights Applied to the new weight matrix W ′ , ensuring that the next calculation uses the latest weights. Using the updated W ′ To calculate C, it reflects the statistical relationship between the features after adjusting the weights.

[0089] The original weight is expressed as:

[0090] W = diagw 11 ,w 22 ,…,w nn

[0091] Where W represents the diagonal weight matrix, which is used to adjust the contribution of each feature in calculating the covariance matrix, w ii Denotes the diagonal elements of the weight matrix W, representing the initial weight of the i-th feature.

[0092] Weight adjustment, expressed as:

[0093]

[0094] Among them, diag means placing the elements in brackets on the diagonal of the matrix. represents the updated weight, α represents the learning rate, a positive scalar value used to control the speed of weight update, ΔE i To represent the performance change index of feature i, the present invention adopts the change of error rate.

[0095] The updated weight matrix W ′ , expressed as:

[0096]

[0097] Among them, W ′ represents the updated diagonal weight matrix.

[0098] The QR algorithm is applied to decompose the weighted covariance matrix C to gradually transform C into a diagonal matrix form. The process involves decomposing C into an orthogonal matrix Q and an upper triangular matrix R, and then iteratively calculating the RQ combination.

[0099] After the QR algorithm converges, the diagonal elements of the upper triangular matrix R, i.e., the eigenvalues ​​of C, are n , which represents the variation of each principal component.

[0100] Calculate the sum of all eigenvalues Represents the total variation in the data.

[0101] Calculate the sum of the first three largest eigenvalues ​​λ1, λ2, and λ3 to evaluate the contribution of these three principal components to the total variation, expressed as:

[0102]

[0103] If this proportion exceeds 75%, the first three principal components are considered to explain more than 75% of the total variation.

[0104] The eigenvalue decomposition is performed after the weighted covariance matrix C is obtained by using the QR algorithm.

[0105] The principal components that maximize the capture of variation in the data include selecting the K-means algorithm for clustering if the first three principal components explain more than 75% of the total variation and the components are dominated by numerical features;

[0106] If the numerical features are not dominant or the first three principal components fail to explain more than 75% of the data variation, analyze the relationship between the data features and use the scatter plot matrix and correlation coefficient matrix to evaluate the linear and nonlinear relationships between the data features and the overall distribution complexity of the features;

[0107] If the metric of the nonlinear relationship between data features exceeds the preset threshold, and the uniform distribution in the feature space exceeds the set consistency standard threshold, the spectral clustering algorithm is used. Spectral clustering is suitable for processing situations where the complex relationship between data features is significant and the data distribution is uniform, and can effectively identify data clusters in complex patterns.

[0108] If the point density is higher than two standard deviations of the average density of the overall data points, and the number of noise points in the data accounts for more than 5% of the total number of data points, the DBSCAN algorithm is selected. The DBSCAN algorithm is superior to data clustering in uneven density and high noise environments, and can identify irregularly shaped clusters and effectively isolate noise points;

[0109] If the judgment rules of the spectral clustering algorithm and the DBSCAN algorithm are not met, the hierarchical clustering algorithm is uniformly adopted. Hierarchical clustering does not depend on the predetermined number of clusters. It is suitable for situations where the data structure is unclear or the data variation is low, and provides a step-by-step refined clustering process.

[0110] Generating the identifier set includes performing attribute analysis on each data segment to assess the sensitivity level and expected frequency of data communication in the network environment;

[0111] Based on the analysis results, an identifier is generated that includes information about the network location and security level of the data fragment;

[0112] Bind identifiers to corresponding data fragments to optimize data routing, storage, and secure processing in the network, ensure identification and secure management of data fragments during network transmission and access, and support efficient data transmission and security control.

[0113] The encrypting of each data fragment includes analyzing an identifier of each data fragment, selecting an encryption algorithm according to a sensitivity level provided by the identifier; applying the selected encryption algorithm to encrypt the data fragment, each data fragment being encrypted independently;

[0114] Generate an independent encryption key for each encrypted data fragment, bind each encryption key to the corresponding encrypted data fragment, and integrate all encrypted data fragments and their corresponding encryption keys to form a set of encrypted data fragments.

[0115] The distributed storage of the encrypted data segments includes performing attribute analysis on each encrypted data segment, determining the data volume by analyzing the number of data points in the data segment, and setting a storage policy according to the sensitivity level contained in the identifier associated with the data segment;

[0116] For data fragments whose volume exceeds a predetermined threshold, further sharding is performed to meet the needs of distributed storage;

[0117] A storage node is selected for each data segment based on the location information of the data segment and the expected frequency of data communication.

[0118] Considering the network topology and data access pattern, storage nodes are optimized to reduce access latency and improve data retrieval efficiency.

[0119] A secure communication protocol is used during data transmission to ensure the security of data during transmission to the storage node.

[0120] Data is securely stored on selected nodes, and regular synchronization and integrity checks are performed to ensure data consistency and up-to-dateness.

[0121] The volume threshold is based on a comprehensive consideration of network bandwidth, storage node capacity, and data processing efficiency.

[0122] The data segments that exceed the threshold are split vertically or horizontally. Vertical segmentation is applicable to structured data and is segmented by field; horizontal segmentation is applicable to unstructured data, such as images and videos, and is segmented by data blocks.

[0123] The basis for segmentation is to ensure that each segmented sub-segment can be processed independently and its size does not negatively affect network transmission and storage efficiency.

[0124] Shard execution:

[0125] For each data fragment that needs to be sharded, a sharding algorithm is implemented to subdivide the large data fragment into smaller, more manageable units.

[0126] Ensure that each sharded sub-segment maintains data integrity and readability to avoid data corruption.

[0127] Each newly generated sub-segment is assigned a new identifier. These identifiers reflect the origin of the original data segment and its position in the overall data structure.

[0128] Update the data management system to ensure that the location and access path of each sub-segment are correctly recorded and synchronized.

[0129] Test the sharding results to verify the consistency and integrity of the data and confirm that the sharding operation does not affect the availability and security of the data.

[0130] Example 2

[0131] Reference Figure 2 , as an embodiment of the present invention, provides an identification encryption storage system for a distributed data element cluster, including:

[0132] Data segmentation module, data identification module, data verification module, data encryption module, data storage module and data decryption module;

[0133] The data segmentation module is used to segment the data set into data segments to generate a data segment set;

[0134] The data identification module is used to assign an identifier to each data segment and generate an identifier set;

[0135] The data verification module is used to generate a verification value for each data fragment using a hash algorithm to form a verification value set;

[0136] The data encryption module is used to encrypt each data fragment and generate a set of encrypted data fragments and a corresponding encryption key set;

[0137] The data storage module is used to store the encrypted data fragments in a distributed manner;

[0138] The data decryption module is used to find the corresponding encrypted data segment according to the identifier when receiving a data segment for data retrieval request, obtain the corresponding encryption key, decrypt the data segment, and compare the hash value of the decrypted data with the check value generated before storage, and return the original data.

[0139] Example 3

[0140] An embodiment of the present invention is different from the first two embodiments in that:

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0143] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0144] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0145] Example 4

[0146] As an embodiment of the present invention, a method for identifying and encrypting the storage of a distributed data element cluster is provided. In order to verify the beneficial effects of the present invention, the superiority of the present invention over traditional methods in terms of data security, storage efficiency and data retrieval performance is verified.

[0147] The experiment was conducted using MATLAB simulation software, equipped with a distributed server cluster to simulate different network conditions and storage devices.

[0148] Prepare different types of data fragments, 1000 data fragments for each type of data, and the size of each fragment ranges from 1MB to 100MB.

[0149] The experimental group applied the method proposed in the present invention, including intelligent segmentation of data fragments, identifier allocation, data encryption and distributed storage strategy.

[0150] The control group applied the traditional data storage method, that is, no intelligent segmentation and encryption strategy selection were performed, and AES encryption and random storage were used uniformly.

[0151] Apply encryption and storage policies to each piece of data. Simulate data storage and retrieval operations in a distributed storage environment. Record the execution time, success rate, and required resources for each operation.

[0152] The data of each method on each performance indicator are collected. The performance of the two methods is compared using statistical analysis methods. The experimental results are shown in Table 1.

[0153] Table 1 Comparison of experimental results

[0154]

[0155] The dynamic data encryption strategy and refined data access control adopted in the present invention resulted in an 18% higher successful defense rate for the experimental group. Data security was enhanced by assigning an independent encryption key and detailed identifier to each data fragment.

[0156] Because the data segmentation strategy allows the system to more accurately determine where and how data is stored, by intelligently segmenting data fragments, storage space can be used more efficiently, reducing unnecessary data duplication and space waste, and ultimately the storage cost of the experimental group was reduced by 60%.

[0157] The optimization of data intelligent partitioning and identifier system has reduced the retrieval time by half. The rationalization of data partitioning ensures that specific data fragments can be quickly located during data retrieval, reducing query time and improving response speed.

[0158] The present invention not only improves the security and efficiency of the entire system by intelligently partitioning the data set and accurately managing the data fragments, but also optimizes the data management structure, making data storage and retrieval operations more efficient. Especially in a distributed environment, this method has outstanding advantages, helping to reduce network load and increase data processing speed.

[0159] Data security is enhanced by using different encryption strategies for each data fragment according to its sensitivity level. The most appropriate storage node is selected based on the access frequency and location information of the data fragment, effectively reducing the delay and cost of data retrieval. Data storage is made more flexible and efficient by dynamically analyzing the characteristics of the data fragment and adjusting the storage strategy accordingly. Regular checks are performed using the generated hash check value to ensure that the integrity and consistency of the encrypted data are not damaged.

[0160] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for storing the identification encryption of a distributed data element cluster, characterized in that: include: Divide the data set into data segments and generate a set of data segments; Generating a set of data segments includes identifying data types in a data set and performing feature extraction; the data types include text data, numerical data, time series data, and image data; Integrate the features extracted from each data type into a comprehensive feature vector; Use principal component analysis to evaluate the main components of the feature vector, quantify the principal component analysis results, select a clustering algorithm, perform clustering operations based on the selected clustering algorithm, split the data into clusters with similar characteristics, and automatically construct each cluster into data segments; Assign an identifier to each data fragment and generate an identifier set; Use a hash algorithm to generate a check value for each data fragment to form a check value set; Encrypt each data fragment to generate a set of encrypted data fragments and a corresponding encryption key set; Distribute and store the encrypted data fragments; When receiving a data retrieval request for an identifier of a data segment, the corresponding encrypted data segment is searched according to the identifier, the corresponding encryption key is obtained, the data segment is decrypted, and the hash value of the decrypted data is compared with the checksum generated before storage, and the original data is returned; The feature extraction includes identifying the language complexity, text length, key topics or sentiment tendency of the text data, and using NLP technology to extract semantic features and sentiment analysis results; Identify the statistical distribution range, standard deviation, kurtosis and skewness of numerical data, and calculate statistical descriptors, including mean, median, and variance; Identify the periodicity, trend, and seasonal changes of time series data and extract Fourier transform features; Identify the resolution, color depth, and content features of image data, and use convolutional neural networks to extract color histograms; The selection clustering algorithm includes standardizing the comprehensive feature vector, calculating the covariance matrix between the features based on the standardized features, and using the covariance matrix to identify the principal component that maximizes the capture of the variation in the data using eigenvalue decomposition; The standardization process is expressed as, Among them, Z i represents the i-th feature after standardization, X i represents the i-th eigenvalue in the original data, μ i represents the mean of the i-th feature, σi represents the standard deviation of the i-th feature, β i represents the adjustment coefficient dynamically calculated based on the coefficient of variation of feature i, σ i is the standard deviation of feature i, μ i is the mean of feature i; The covariance matrix is ​​expressed as, Where C represents the weighted covariance matrix, W represents the diagonal weight matrix, and Z k represents the standardized feature vector of the kth sample, n represents the number of samples, and T represents the transposition operator; Dynamically adjust the weight of each feature based on real-time data or performance feedback i ; The adjusted weight Apply to the new weight matrix W′ and use the updated W′ to calculate C; The original weight is expressed as: In=diag(in 11 ,In 22 ,...,In nn ) Where W represents the diagonal weight matrix, which is used to adjust the contribution of each feature in calculating the covariance matrix, w ii The diagonal elements of the weight matrix W represent the initial weight of the i-th feature; Weight adjustment, expressed as: Among them, diag means placing the elements in brackets on the diagonal of the matrix. represents the updated weight, α represents the learning rate, a positive scalar value used to control the speed of weight update, ΔE j The performance change index of feature i is represented by the change of error rate in the present invention; The updated weight matrix W′ is expressed as: Where W′ represents the updated diagonal weight matrix; The eigenvalue decomposition is performed using the QR algorithm after the weighted covariance matrix C is obtained; Apply the QR algorithm to decompose the weighted covariance matrix C to gradually transform c into a diagonal matrix form. The process involves decomposing C into an orthogonal matrix Q and an upper triangular matrix R, and then iteratively calculating the RQ combination; After the QR algorithm converges, the diagonal elements of the upper triangular matrix R, i.e., the eigenvalues ​​of C, are n , represents the variation of each principal component; Calculate the sum of all eigenvalues represents the total variation of the data; Calculate the sum of the first three largest eigenvalues ​​λ1, λ2, and λ3 to evaluate the contribution of these three principal components to the total variation, expressed as: If this proportion exceeds 75%, it is considered that the first three principal components explain more than 75% of the total variation; The principal components that maximize the capture of variation in the data include selecting the K-means algorithm for clustering if the first three principal components explain more than 75% of the total variation and the components are dominated by numerical features; If the numerical features are not dominant or the first three principal components fail to explain more than 75% of the data variation, analyze the relationship between the data features and use the scatter plot matrix and correlation coefficient matrix to evaluate the linear and nonlinear relationships between the data features and the overall distribution complexity of the features; If the metric of the nonlinear relationship between data features exceeds the preset threshold, and the uniform distribution degree in the feature space exceeds the set consistency standard threshold, the spectral clustering algorithm is used; If the point density is higher than two standard deviations of the average density of the overall data points, and the number of noise points in the data accounts for more than 5% of the total number of data points, the DBSCAN algorithm is selected; If the data feature relationship analysis does not show significant nonlinear relationships or uneven density, and the number of noise points does not meet the criteria for using the DBSCAN algorithm, the hierarchical clustering algorithm is used.

2. The method for storing the identification encryption of a distributed data element cluster as claimed in claim 1, characterized in that: Generating the identifier set includes performing attribute analysis on each data segment to assess the sensitivity level and expected frequency of data communication in the network environment; Based on the analysis results, an identifier is generated that includes information about the network location and security level of the data fragment; Bind identifiers to corresponding data fragments to optimize data routing, storage, and secure processing in the network.

3. The method for storing the identification encryption of a distributed data element cluster as claimed in claim 2, characterized in that: The encrypting of each data fragment includes analyzing an identifier of each data fragment, selecting an encryption algorithm according to a sensitivity level provided by the identifier; applying the selected encryption algorithm to encrypt the data fragment, each data fragment being encrypted independently; Generate an independent encryption key for each encrypted data fragment, bind each encryption key to the corresponding encrypted data fragment, and integrate all encrypted data fragments and their corresponding encryption keys to form a set of encrypted data fragments.

4. The method for storing the identification encryption of a distributed data element cluster as claimed in claim 3, characterized in that: The distributed storage of the encrypted data segments includes performing attribute analysis on each encrypted data segment, determining the data volume by analyzing the number of data points in the data segment, and setting a storage policy according to the sensitivity level contained in the identifier associated with the data segment; For data fragments whose volume exceeds a predetermined threshold, further sharding is performed to meet the needs of distributed storage; A storage node is selected for each data segment based on the location information of the data segment and the expected frequency of data communication.

5. A system using the identification encryption storage method of a distributed data element cluster as described in any one of claims 1 to 4, characterized in that: include: Data segmentation module, data identification module, data verification module, data encryption module, data storage module and data decryption module; The data segmentation module is used to segment the data set into data segments to generate a data segment set; The data identification module is used to assign an identifier to each data segment and generate an identifier set; The data verification module is used to generate a verification value for each data fragment using a hash algorithm to form a verification value set; The data encryption module is used to encrypt each data fragment and generate a set of encrypted data fragments and a corresponding encryption key set; The data storage module is used to store the encrypted data fragments in a distributed manner; The data decryption module is used to find the corresponding encrypted data segment according to the identifier when receiving a data segment for data retrieval request, obtain the corresponding encryption key, decrypt the data segment, and compare the hash value of the decrypted data with the check value generated before storage, and return the original data.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the processor implements the steps of the identification encryption storage method of the distributed data element cluster described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the identification encryption storage method of a distributed data element cluster described in any one of claims 1 to 4 are implemented.

Images