Unstructured data storage optimization method and system based on information entropy and block chain

Abstract

The invention provides an unstructured data storage optimization method and system based on information entropy and a block chain, and relates to the technical field of data storage, and the method comprises the steps: building a block chain storage network to obtain data, recognizing redundant data through dimension reduction, mapping and hyperplane construction, and removing the redundant data based on fuzzy similarity. The to-be-stored data is partitioned, the access frequency is calculated, the consistent Hash algorithm is executed to determine the storage position, the access index and the verification sequence are established, and the data integrity verification is executed, so that the data access efficiency can be improved, and the storage security can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to an unstructured data storage optimization method and system based on information entropy and blockchain. Background Art

[0002] Unstructured data includes images, videos, text, audio, and other forms. Traditional data storage systems face problems such as low storage efficiency, poor security, and complex management when processing large amounts of unstructured data. The distributed, tamper-proof, and traceable characteristics of blockchain technology provide new ideas and methods for storing unstructured data. However, existing unstructured data storage technologies have difficulty in effectively identifying and processing redundant information in unstructured data, resulting in a large amount of duplicate or similar data being stored repeatedly. This not only wastes storage resources, but also increases the complexity of data management. There is a lack of effective mechanisms for data distribution and access optimization, and it is impossible to intelligently allocate data based on access frequency and characteristics, resulting in congestion in hot data access and uneven resource utilization, which reduces the performance and response speed of the entire storage system and makes it difficult to ensure the security and reliability of unstructured data in a distributed environment. In particular, there is a lack of efficient data access verification and integrity proof methods, which exposes the system to the risk of data tampering and unauthorized access. Summary of the Invention

[0003] The embodiments of the present invention provide a method and system for optimizing unstructured data storage based on information entropy and blockchain, which can at least solve some of the problems existing in the prior art.

[0004] A first aspect of an embodiment of the present invention provides an unstructured data storage optimization method based on information entropy and blockchain, comprising: Establish a blockchain storage network and obtain unstructured data, construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain dimensionality-reduced data; The reduced-dimensional data is mapped to a high-dimensional feature space using a preset kernel function, an optimal classification hyperplane is constructed, a distance vector from each reduced-dimensional data to the optimal classification hyperplane is calculated to obtain an initial redundant data set, a binary coding sequence is generated based on the data features in the initial redundant data set, selection, crossover, and mutation operations are performed on the binary coding sequence to obtain an optimized feature sequence, redundant data is labeled according to the optimized feature sequence, fuzzy similarity of different redundant data is calculated, and data with a fuzzy similarity greater than a preset fuzzy similarity threshold is eliminated to obtain a data set to be stored; Dividing the data set to be stored into blocks according to a preset size to obtain a data block set, calculating the access frequency of each data block in the data block set, executing a consistent hashing algorithm based on the access frequency to allocate the data block set to obtain a data block storage location table, establishing a data access index based on the data block storage location table, performing segmented encoding on each data block and calculating a hash value to obtain a data block feature table, and generating a data access verification sequence based on the data block feature table and the data access index; Perform data integrity verification according to the data access verification sequence, generate a certificate containing data identification and signature, and record the verification result to the blockchain.

[0005] In an optional embodiment, Establish a blockchain storage network and obtain unstructured data. Construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain the reduced dimensionality data including: Obtain unstructured data and establish a blockchain storage network consisting of multiple nodes; Extracting frequency features and time series features from the unstructured data, calculating feature vectors based on the frequency features and the time series features, and constructing a covariance matrix based on the feature vectors; Performing eigenvalue decomposition on the covariance matrix to obtain initial eigenvalues, reconstructing the unstructured data based on the initial eigenvalues ​​to obtain reduced-dimensionality data, and storing the reduced-dimensionality data in the blockchain storage network.

[0006] In an optional embodiment, The reduced-dimensional data are mapped to a high-dimensional feature space using a preset kernel function, an optimal classification hyperplane is constructed, a distance vector from each reduced-dimensional data to the optimal classification hyperplane is calculated to obtain an initial redundant data set, a binary code sequence is generated based on the data features in the initial redundant data set, and selection, crossover, and mutation operations are performed on the binary code sequence to obtain an optimized feature sequence, including: Obtaining dimensionality-reduced data, calculating the Euclidean distance between different data points in the dimensionality-reduced data, constructing a Gaussian kernel function based on the Euclidean distance and a preset kernel function bandwidth parameter, and mapping the dimensionality-reduced data to a high-dimensional feature space using the Gaussian kernel function to obtain high-dimensional feature data; Constructing an optimization objective function based on the high-dimensional feature data, wherein the optimization objective function includes a normal vector modulus minimization term and a constraint relaxation term weighted by a preset penalty factor, and solving the optimization objective function to obtain a normal vector and a bias term of an optimal classification hyperplane; Calculating a distance value from each data point in the dimensionality reduction data to the optimal classification hyperplane based on the normal vector and the bias term, sorting the distance values ​​from small to large, selecting the data points in the top 10% to construct an initial redundant data set, extracting a feature vector corresponding to each data point from the initial redundant data set, calculating the mean of feature components of each dimension of the feature vector, comparing the feature components with the mean of the corresponding dimension, assigning a binary value of 1 to feature components greater than the mean, and assigning a binary value of 0 to feature components less than the mean, to generate a binary code sequence; The fitness value of each sequence in the binary code sequence is calculated, a roulette wheel selection operation is performed based on the fitness value to construct a high-quality sequence, a single-point crossover operation is performed on the high-quality sequence to generate a crossover sequence, and a random inversion operation is performed on the sites of the crossover sequence to obtain an optimized feature sequence.

[0007] In an optional embodiment, Redundant data are marked according to the optimized feature sequence, fuzzy similarities of different redundant data are calculated, and data with fuzzy similarities greater than a preset fuzzy similarity threshold are eliminated to obtain a data set to be stored, including: Obtaining an optimized feature sequence and mapping each feature dimension in the optimized feature sequence into a fuzzy membership value, constructing a triangular membership function based on the fuzzy membership value, and dynamically adjusting parameters of the triangular membership function based on data distribution characteristics corresponding to the unstructured data to obtain an adaptive fuzzy feature; Building a multidimensional fuzzy rule base based on the adaptive fuzzy feature, assigning an initial weight coefficient to each rule in the multidimensional fuzzy rule base, calculating the fuzzy similarity between different data according to the multidimensional fuzzy rule base, and marking data pairs with fuzzy similarity greater than a preset fuzzy similarity threshold as pre-screened redundant data; Calculating the local density value corresponding to the primary screening redundant data based on the distance between data points and the cutoff distance parameter, fine-screening the primary screening redundant data according to the local density value, and marking the data with a local density value greater than a preset density threshold as candidate redundant data; The integrity scores of the candidate redundant data are calculated, candidate redundant data with integrity scores lower than a preset integrity threshold are eliminated, and a data set to be stored is constructed based on the retained candidate redundant data.

[0008] In an optional embodiment, Dividing the data set to be stored into blocks according to a preset size to obtain a data block set, calculating the access frequency of each data block in the data block set, executing a consistent hashing algorithm to allocate the data block set to obtain a data block storage location table, and establishing a data access index based on the data block storage location table includes: Preliminarily dividing the data set to be stored into blocks according to a preset size, calculating an optimal block factor including the total size of the data set and the data density variance, and adaptively dividing the data set to be stored into blocks according to the optimal block factor to obtain a data block set; For each data block in the data block set, calculating the access frequency of each data block in the data block set based on the number of historical accesses within a preset time window and a preset time decay factor; Calculating a storage node weight value based on the access frequency and a basic weight value corresponding to each storage node, performing virtual node expansion on the storage node based on the storage node weight value to obtain a virtual node distribution sequence, calculating a data block hash value for each data block in the data block set, and executing a consistent hashing algorithm to map a weighted sum of the data block hash value and the access frequency to a hash space corresponding to the virtual node distribution sequence to obtain a data block storage location table; Based on the data block storage location table, a main index table is established to record the mapping relationship between data blocks and physical nodes. Based on the access frequency, an access frequency threshold is set and a secondary index table for hot data access is established. The physical distance and logical distance are calculated, the routing weight is constructed and a routing table is generated. The main index table, the secondary index table and the routing table are combined to form a data access index.

[0009] In an optional embodiment, Segment encoding is performed on each data block, and a hash value is calculated to obtain a data block feature table. Generating a data access verification sequence according to the data block feature table and the data access index includes: Obtain historical access records of data blocks and generate access pattern time series based on the historical access records; The content entropy, boundary sensitivity, and data density of the data block are calculated to construct a data block content feature vector. The optimal segment size is calculated based on the information entropy of the data block content feature vector and the variance of the access pattern time series. Each data block is segmented according to the optimal segment size to obtain a segment sequence. Obtain historical error records of data blocks and calculate the data block error rate; calculate the number of information symbols based on the access frequency of the data blocks in the segmented sequence; calculate the number of check symbols based on the data block error rate and the number of information symbols; encode the segmented sequence to obtain a coded block set; obtain a preset minimum segment size of the data block; calculate the tree structure depth based on the data block size and construct a tree structure; assign node numbers to the coded blocks in the coded block set to generate parent node pointers; calculate a hierarchical hash value for the coded block set; and form a data block feature table with the hierarchical hash value, tree structure depth information, and parent node pointers; The access frequency of the data block in the data access index is obtained to calculate the cache threshold, and the data nodes with access frequency greater than the cache threshold are cached and pre-fetched to establish a verification path. The current system time is obtained as the timestamp, and the read and write status, integrity status and access permission status of the data block are obtained to form the node status information. The hierarchical hash value, timestamp and node status information are signed to obtain a verification tag, and a data access verification sequence is generated based on the verification tag, verification path and data block feature table.

[0010] In an optional embodiment, Performing data integrity verification according to the data access verification sequence, generating a certificate containing data identification and signature, and recording the verification results to the blockchain includes: Obtaining a data access verification sequence, recursively verifying the hierarchical hash value based on the verification path in the data access verification sequence, and generating a verification result; Generate a data identifier based on the verification result, sign the data identifier to obtain an integrity signature, and combine the data identifier and the integrity signature to form a verification certificate; Submit the verification proof to the blockchain network and record it.

[0011] A second aspect of an embodiment of the present invention provides an unstructured data storage optimization system based on information entropy and blockchain, including: The first unit is used to establish a blockchain storage network and obtain unstructured data, construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain dimensionality-reduced data; The second unit is configured to map the reduced-dimensional data to a high-dimensional feature space using a preset kernel function, construct an optimal classification hyperplane, calculate a distance vector from each reduced-dimensional data to the optimal classification hyperplane to obtain an initial redundant data set, generate a binary code sequence based on data features in the initial redundant data set, perform selection, crossover, and mutation operations on the binary code sequence to obtain an optimized feature sequence, label redundant data according to the optimized feature sequence, calculate fuzzy similarity of different redundant data, and eliminate data whose fuzzy similarity is greater than a preset fuzzy similarity threshold to obtain a data set to be stored; A third unit is configured to divide the data set to be stored into blocks according to a preset size to obtain a data block set, calculate the access frequency of each data block in the data block set, execute a consistent hashing algorithm to allocate the data block set based on the access frequency to obtain a data block storage location table, establish a data access index based on the data block storage location table, perform segmented encoding on each data block and calculate a hash value to obtain a data block feature table, and generate a data access verification sequence based on the data block feature table and the data access index; The fourth unit is used to perform data integrity verification according to the data access verification sequence, generate a certificate including the data identification and signature, and record the verification result to the blockchain.

[0012] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0013] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0014] In the present invention, by constructing a covariance matrix and performing eigenvalue decomposition in the blockchain network, effective dimensionality reduction processing of unstructured data is achieved, data redundancy and storage space occupancy are reduced, and data processing efficiency is improved. The fuzzy similarity calculation mechanism and genetic algorithm are introduced to optimize the feature sequence, effectively identifying and eliminating redundant data with high similarity, while ensuring data integrity. The blockchain storage structure is significantly optimized, avoiding the problem of a large amount of duplicate data occupying storage resources. An efficient data access index system is constructed by combining data block access frequency and consistent hashing algorithm, and data integrity verification is achieved through data block segmentation coding and hash calculation, ensuring the security and reliability of unstructured data in the blockchain storage network, while improving data query and access efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of an unstructured data storage optimization method based on information entropy and blockchain according to an embodiment of the present invention; Figure 2 This is a flowchart of data block verification and processing of an unstructured data storage optimization method based on information entropy and blockchain in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0017] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0018] Figure 1 This is a flow chart of an unstructured data storage optimization method based on information entropy and blockchain according to an embodiment of the present invention. Figure 1 As shown, the method includes: Establish a blockchain storage network and obtain unstructured data, construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain dimensionality-reduced data; The reduced-dimensional data is mapped to a high-dimensional feature space using a preset kernel function, an optimal classification hyperplane is constructed, a distance vector from each reduced-dimensional data to the optimal classification hyperplane is calculated to obtain an initial redundant data set, a binary coding sequence is generated based on the data features in the initial redundant data set, selection, crossover, and mutation operations are performed on the binary coding sequence to obtain an optimized feature sequence, redundant data is labeled according to the optimized feature sequence, fuzzy similarity of different redundant data is calculated, and data with a fuzzy similarity greater than a preset fuzzy similarity threshold is eliminated to obtain a data set to be stored; Dividing the data set to be stored into blocks according to a preset size to obtain a data block set, calculating the access frequency of each data block in the data block set, executing a consistent hashing algorithm based on the access frequency to allocate the data block set to obtain a data block storage location table, establishing a data access index based on the data block storage location table, performing segmented encoding on each data block and calculating a hash value to obtain a data block feature table, and generating a data access verification sequence based on the data block feature table and the data access index; Perform data integrity verification according to the data access verification sequence, generate a certificate containing data identification and signature, and record the verification result to the blockchain.

[0019] In an optional embodiment, Establish a blockchain storage network and obtain unstructured data. Construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain the reduced dimensionality data including: Obtain unstructured data and establish a blockchain storage network consisting of multiple nodes; Extracting frequency features and time series features from the unstructured data, calculating feature vectors based on the frequency features and the time series features, and constructing a covariance matrix based on the feature vectors; Performing eigenvalue decomposition on the covariance matrix to obtain initial eigenvalues, reconstructing the unstructured data based on the initial eigenvalues ​​to obtain reduced-dimensionality data, and storing the reduced-dimensionality data in the blockchain storage network.

[0020] Acquire unstructured data from various data sources, such as text documents, images, audio, and video. For text data, web crawlers can be used to capture it from designated websites; for audio data, microphones can be used to capture it; and for video data, cameras can be used to record it. The acquired unstructured data is stored in temporary storage for subsequent processing.

[0021] Establish a blockchain storage network composed of multiple nodes, each equipped with data storage, verification, and synchronization capabilities. The blockchain network utilizes a distributed architecture, with nodes communicating via a peer-to-peer (P2P) protocol. Each node maintains a complete blockchain ledger to ensure data consistency. The block structure consists of a header and a body. The header contains the hash value of the previous block, a timestamp, a difficulty target, and a random number; the body contains transaction data, which is the reduced unstructured data in this technical solution. A consensus algorithm is used between nodes to ensure data consistency. Depending on actual needs, algorithms such as proof-of-work, proof-of-stake, or practical Byzantine fault tolerance can be selected.

[0022] Extracting frequency features from unstructured data involves analyzing the frequency of occurrence of each component element in the data. For text data, word or character frequency can be calculated; for image data, pixel value distribution can be analyzed; and for audio data, the energy distribution of each frequency band can be calculated. For text data, for example, word segmentation techniques are used to segment the text into sequences of words, and the frequency of each word in the document is calculated. The dimensionality of the frequency feature vector is comparable to the size of the vocabulary, with each dimension representing the frequency of the corresponding word. In practical applications, it is necessary to filter out low-frequency words and retain only the features of high-frequency words, thereby reducing the vector dimensionality.

[0023] Extracting time series features primarily focuses on how data elements change over time. For time series data, features such as trends, periodicity, and autocorrelation coefficients can be extracted. For text data, transitions between sentences or paragraphs can be analyzed. For video data, inter-frame variation can be extracted. For time series data, for example, statistics such as mean, variance, kurtosis, and skewness can be calculated within different time windows to form a time series feature vector.

[0024] Based on the extracted frequency and time series features, feature vectors are calculated and fused. Feature concatenation can be used to directly concatenate the frequency and time series feature vectors to form a comprehensive feature vector. Alternatively, a weighted fusion approach can be used to assign different weights to the two feature types before fusing them. To improve feature representation, feature transformation techniques such as logarithmic transformation and normalization can be introduced. After feature vector calculation, each unstructured data sample corresponds to a feature vector of fixed dimension.

[0025] The eigenvectors of all samples are combined into a feature matrix, and the covariance relationships between features are calculated. Each element of the covariance matrix represents the correlation between two features. During the calculation process, the eigenvectors are first centered. This means that the mean of each feature is subtracted so that the center of the feature distribution is at the origin. After centering, the covariance between the features is calculated and the covariance matrix is ​​populated. The covariance matrix is ​​a symmetric matrix, with the diagonal elements representing the variance of each feature and the off-diagonal elements representing the covariance between pairs of features.

[0026] By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, we can identify the main directions of variation in the data. The magnitude of the eigenvalue indicates the degree of data variation in the direction of the corresponding eigenvector; larger eigenvalues ​​indicate more significant data variation in that direction. In practice, numerical methods such as power iteration and QR decomposition can be used to determine the eigenvalues ​​and eigenvectors.

[0027] Sorting by eigenvalues, the top eigenvectors are selected to form a dimensionality reduction projection matrix. The number of columns in the projection matrix determines the dimensions of the data after dimensionality reduction. The number of dimensions retained can be determined based on the cumulative variance contribution. For example, the top K eigenvectors are selected so that the sum of their corresponding eigenvalues ​​accounts for at least 95% of the total eigenvalue sum.

[0028] The original eigenvectors are transformed using a projection matrix to obtain a reduced-dimensional representation of the data. This reduced-dimensional data not only retains the essential information of the original data but also significantly reduces storage space requirements. For example, 100MB of text data might require only 20MB of storage space after applying this method, significantly improving storage efficiency.

[0029] The reduced-dimensionality data is stored on the blockchain network, packaged into transactions, verified by network nodes, and added to blocks. To improve storage efficiency, the reduced-dimensionality data can be further compressed or sharded. Each data block is appended with metadata such as a timestamp and data hash to facilitate subsequent retrieval and verification. Once the data is stored, network nodes automatically synchronize the ledger to ensure data consistency across the entire network.

[0030] In this embodiment, after obtaining unstructured data, by extracting the frequency features and time series features therein and calculating the eigenvectors, the dynamic change patterns and internal correlations of the unstructured data can be effectively characterized, avoiding the high-dimensional redundancy problem caused by directly processing the original data. By constructing a covariance matrix and performing eigenvalue decomposition, the most representative important feature dimensions in the data can be identified, and the reduced-dimensional data can be stored in a blockchain storage network composed of multiple nodes. This can realize decentralized storage of data in a distributed environment, ensure that the data cannot be tampered with and is traceable, and improve data security and credibility. The multi-node collaborative storage mechanism based on the blockchain can also improve the stability and disaster recovery capabilities of data storage and access, so that unstructured data has higher utilization value under the dual protection of efficient compression and secure storage.

[0031] In an optional embodiment, The reduced-dimensional data are mapped to a high-dimensional feature space using a preset kernel function, an optimal classification hyperplane is constructed, a distance vector from each reduced-dimensional data to the optimal classification hyperplane is calculated to obtain an initial redundant data set, a binary coding sequence is generated based on the data features in the initial redundant data set, and selection, crossover, and mutation operations are performed on the binary coding sequence to obtain an optimized feature sequence, including: Obtaining dimensionality-reduced data, calculating the Euclidean distance between different data points in the dimensionality-reduced data, constructing a Gaussian kernel function based on the Euclidean distance and a preset kernel function bandwidth parameter, and mapping the dimensionality-reduced data to a high-dimensional feature space using the Gaussian kernel function to obtain high-dimensional feature data; Constructing an optimization objective function based on the high-dimensional feature data, wherein the optimization objective function includes a normal vector modulus minimization term and a constraint relaxation term weighted by a preset penalty factor, and solving the optimization objective function to obtain a normal vector and a bias term of an optimal classification hyperplane; Calculating a distance value from each data point in the dimensionality reduction data to the optimal classification hyperplane based on the normal vector and the bias term, sorting the distance values ​​from small to large, selecting the data points in the top 10% to construct an initial redundant data set, extracting a feature vector corresponding to each data point from the initial redundant data set, calculating the mean of feature components of each dimension of the feature vector, comparing the feature components with the mean of the corresponding dimension, assigning a binary value of 1 to feature components greater than the mean, and assigning a binary value of 0 to feature components less than the mean, to generate a binary code sequence; The fitness value of each sequence in the binary code sequence is calculated, a roulette wheel selection operation is performed based on the fitness value to construct a high-quality sequence, a single-point crossover operation is performed on the high-quality sequence to generate a crossover sequence, and a random inversion operation is performed on the sites of the crossover sequence to obtain an optimized feature sequence.

[0032] Obtain reduced-dimensionality data. Reduced-dimensionality data can be a low-dimensional representation extracted from raw unstructured data using methods such as principal component analysis, linear discriminant analysis, or autoencoders. In practical applications, reduced-dimensionality data may be a numerical vector of unstructured data such as text, images, or audio, after feature extraction and dimensionality reduction. For example, a 1,000-word text might be converted into a 50-dimensional vector after word frequency counting and dimensionality reduction. Each element in the vector represents the projection of the original text along a specific feature direction.

[0033] Calculate the Euclidean distance between different data points in the dimensionality-reduced data. The Euclidean distance is calculated by square the difference between the two data points in the corresponding dimension, summing the squares, and then taking the square root. For example, if two 5-dimensional vectors have values ​​of vector A [0.2, 0.5, 0.8, 0.3, 0.6] and vector B [0.4, 0.3, 0.7, 0.5, 0.2], when calculating the Euclidean distance between them, first find the sum of the squares of the differences in the corresponding positions: (0.2-0.4) 2 +(0.5-0.3) 2 +(0.8-0.7) 2 +(0.3-0.5) 2 +(0.6-0.2)2=0.04+0.04+0.01+0.04+0.16=0.29, and then taking the square root, we get the Euclidean distance of about 0.54.

[0034] A Gaussian kernel function is constructed based on the Euclidean distance and a preset kernel bandwidth parameter to achieve nonlinear data transformation. The Gaussian kernel function essentially maps the Euclidean distance to a similarity value between 0 and 1 using an exponential function. The kernel bandwidth parameter controls the shape of the Gaussian function. A smaller bandwidth parameter makes the function curve steeper and more sensitive to distance changes. In practical applications, the bandwidth parameter can be determined through cross-validation, for example, by setting candidate values ​​within a range of 0.1 to 10 and selecting the value that optimizes classification performance. Assuming a bandwidth parameter of 2, the Gaussian kernel function value between two data points with a distance of 0.54 is an exponential function with the natural logarithm as the base. The square of the distance with a negative exponent divided by twice the square of the bandwidth parameter is calculated to be approximately 0.93, indicating a high degree of similarity between the two points in high-dimensional space.

[0035] Using a Gaussian kernel function to map the reduced-dimensional data to a higher-dimensional feature space eliminates the need to explicitly compute the high-dimensional features, instead implicitly achieving this through kernel techniques. Data points in the higher-dimensional feature space are more linearly separable, facilitating subsequent classification tasks. After data mapping, data points that were originally difficult to separate linearly in the lower-dimensional space may become linearly separable in the higher-dimensional space.

[0036] An optimization objective function is constructed based on high-dimensional feature data. This function consists of a normal vector modulus minimization term and a constraint relaxation term weighted by a pre-set penalty factor. The normal vector modulus minimization term aims to maximize the classification margin, while the constraint relaxation term allows some samples to violate the classification constraints, thereby improving the model's generalization ability. The penalty factor is a trade-off parameter that controls the model's tolerance for misclassified samples. A larger penalty factor prioritizes reducing misclassifications, while a smaller penalty factor prioritizes maximizing the margin. In practice, the penalty factor can be set between 0.1 and 10, with the optimal value determined through grid search.

[0037] Solving the optimization objective function yields the normal vector and bias term of the optimal classification hyperplane. Using a sequential minimization algorithm, the normal vector and bias term are iteratively optimized using Lagrange multipliers. The normal vector determines the direction of the hyperplane, while the bias term determines its position. For example, for two-dimensional data, if the normal vector is [0.8, 0.6] and the bias term is -0.5, the classification hyperplane equation is 0.8x + 0.6y - 0.5 = 0, where x and y are the two characteristic components of the data point.

[0038] The distance between each data point in the dimensionality-reduced data and the optimal classification hyperplane is calculated based on the normal vector and the bias term. The distance is calculated by adding the bias term to the dot product of the data point's eigenvector and the normal vector, then dividing by the modulus of the normal vector. The sign of the distance value indicates which side of the hyperplane the data point lies on, and the absolute value represents the perpendicular distance from the data point to the hyperplane. Data points with distance values ​​close to zero are located near the decision boundary, tend to contain more information, and have a greater impact on the classification results.

[0039] The distance values ​​are sorted from smallest to largest, and the top 10% of data points are selected to form the initial redundant dataset. Data points closest to the decision boundary are considered candidates with high information redundancy. In the case of 10,000 data points, the 1,000 points with the smallest distances are selected to form the initial redundant dataset.

[0040] Extract the feature vector corresponding to each data point from the initial redundant dataset and calculate the mean of the feature components in each dimension of the feature vector. For a 50-dimensional feature vector, calculate the mean value of 1000 data points in each dimension to obtain a 50-dimensional mean vector. Compare the feature components with the mean of the corresponding dimension. Feature components greater than the mean are assigned a binary value of 1, and feature components less than the mean are assigned a binary value of 0, thus generating a binary code sequence.

[0041] The fitness value of each sequence in the binary-coded sequence is calculated. The fitness value reflects the degree of redundancy of the corresponding data points in the sequence. Fitness calculation can be based on information entropy or similarity with other data points. Lower information entropy or higher similarity indicates greater data redundancy and a higher fitness value. Roulette wheel selection is performed based on the fitness values ​​to construct high-quality sequences. Roulette wheel selection allocates selection probability proportionally based on the fitness value, with sequences with higher fitness values ​​having a greater probability of being selected. Out of 1000 sequences, 500 high-quality sequences may be selected for the next stage of processing.

[0042] Perform a single-point crossover on high-quality sequences to generate a crossover sequence. Single-point crossover randomly selects a crossover point between two parent sequences and swaps all bits after that point, generating two new subsequences. For example, if two 50-bit parent sequences are crossed at position 20, two new sequences will be generated, each containing the first 20 bits of one parent sequence and the last 30 bits of the other. Perform a random reversal on the positions in the crossover sequence to generate an optimized signature sequence. Random reversal inverts each bit in the sequence with a certain probability (e.g., 0.01), converting 0 to 1 and vice versa. This increases population diversity and avoids falling into local optima.

[0043] In this embodiment, the Euclidean distance between different data points is calculated and a Gaussian kernel function is constructed in combination with the kernel function bandwidth parameter to map the data to a high-dimensional feature space, effectively improving the separability and discrimination of the original dimensionality-reduced data. The hyperplane is used to calculate the distance from each data point to the classification boundary, and the top 10% of data points close to the boundary are selected as the initial redundant data set, which helps to identify data samples with potential redundancy or blurred boundaries in the feature distribution and improve the targetedness of subsequent optimization processing. By calculating the fitness value of the coding sequence and adopting a roulette wheel selection mechanism, the feature patterns with better performance can be retained. The feature space is further explored by combining single-point crossover and random inversion operations. The generated optimized feature sequence can highlight the optimal feature combination while maintaining diversity, thereby improving the feature representation capability while eliminating redundancy.

[0044] In an optional embodiment, Redundant data are marked according to the optimized feature sequence, fuzzy similarities of different redundant data are calculated, and data with fuzzy similarities greater than a preset fuzzy similarity threshold are eliminated to obtain a data set to be stored, including: Obtaining an optimized feature sequence and mapping each feature dimension in the optimized feature sequence into a fuzzy membership value, constructing a triangular membership function based on the fuzzy membership value, and dynamically adjusting parameters of the triangular membership function based on data distribution characteristics corresponding to the unstructured data to obtain an adaptive fuzzy feature; Building a multidimensional fuzzy rule base based on the adaptive fuzzy feature, assigning an initial weight coefficient to each rule in the multidimensional fuzzy rule base, calculating the fuzzy similarity between different data according to the multidimensional fuzzy rule base, and marking data pairs with fuzzy similarity greater than a preset fuzzy similarity threshold as pre-screened redundant data; Calculating the local density value corresponding to the primary screening redundant data based on the distance between data points and the cutoff distance parameter, fine-screening the primary screening redundant data according to the local density value, and marking the data with a local density value greater than a preset density threshold as candidate redundant data; The integrity scores of the candidate redundant data are calculated, candidate redundant data with integrity scores lower than a preset integrity threshold are eliminated, and a data set to be stored is constructed based on the retained candidate redundant data.

[0045] Obtain an optimized feature sequence. This sequence is typically generated by a genetic algorithm or other optimization algorithm and contains key feature information for unstructured data. This feature sequence may be in binary form, such as "10101100," or in real number form, such as [0.3, 0.7, 0.2, 0.9]. When mapping each feature dimension in the optimized feature sequence to a fuzzy membership value, the distribution range and semantic meaning of the feature value must be considered. For binary features, 0 can be directly mapped to 0.0 and 1 to 1.0. For continuous features, normalization can be used to map them to the range [0, 1]. For example, for a feature value of 15 in the range [-10, 30], linear normalization can yield a membership value of (15 - (-10)) / (30 - (-10)) = 0.625.

[0046] A triangular membership function is constructed based on the fuzzy membership values. The triangular membership function is defined by three parameters: the left endpoint, the vertex, and the right endpoint. For example, if the membership value for a certain feature dimension is 0.6, the triangular membership function can be initially set to 0.4 for the left endpoint, 0.6 for the vertex, and 0.8 for the right endpoint. This means that when the eigenvalue is exactly 0.6, the membership is 1; when the eigenvalue deviates from 0.6 but is within the range [0.4, 0.8], the membership decreases linearly with increasing distance; and when the eigenvalue is less than 0.4 or greater than 0.8, the membership is 0.

[0047] Dynamically adjust the parameters of the triangular membership function based on the data distribution characteristics of unstructured data to make fuzzy reasoning more consistent with the actual data distribution. Adjustment methods can be based on the statistical characteristics of the data, such as mean, variance, and skewness. For uniformly distributed features, the three parameters of the triangular membership function can be set to even intervals. For skewed features, the triangular membership function can be offset toward areas with dense data. For example, if a feature has dense data points within the range [0.5, 0.7], the left endpoint, vertex, and right endpoint of the triangular membership function can be adjusted to 0.45, 0.6, and 0.75, respectively, to better cover high-density areas.

[0048] A multidimensional fuzzy rule base is constructed based on adaptive fuzzy features. Multidimensional fuzzy rules adopt an "if...then..." format to describe the degree of data redundancy under different feature combinations. For data with n feature dimensions, theoretically, 3^n rules can be constructed, but in practice, only important rules are retained to reduce computational complexity. For example, in a two-dimensional feature space, nine rules can be constructed: if feature 1 is low and feature 2 is low, then the redundancy is high; if feature 1 is low and feature 2 is medium, then the redundancy is medium, and so on. The specific form of the rules depends on the semantic meaning and distribution characteristics of the data.

[0049] Each rule in the multidimensional fuzzy rule base is assigned an initial weight coefficient to reflect its importance. Initial weights can be based on domain knowledge or statistically derived from training data. For example, for text data, rules describing keyword frequency may have a higher weight; for image data, rules describing texture features may be more important. Initial weights can be set in the range [0, 1]. For example, assigning weights of [0.9, 0.7, 0.5, 0.7, 0.5, 0.3, 0.5, 0.3, 0.1] to each of the nine rules indicates the importance of different rule combinations.

[0050] Calculating fuzzy similarity between different data points using a multidimensional fuzzy rule base requires comprehensive consideration of the trigger strength and weight of each rule. The calculation process involves calculating the trigger strength of each rule for each pair of data points; multiplying the trigger strength by the rule weight to obtain the weighted trigger strength; and summing the weighted trigger strengths of all rules and dividing it by the sum of the weights to obtain the normalized fuzzy similarity. Fuzzy similarity ranges from 0 to 1, with larger values ​​indicating greater similarity between two data points.

[0051] Data pairs with fuzzy similarity greater than a preset fuzzy similarity threshold are marked as pre-screened redundant data. The fuzzy similarity threshold setting depends on the redundancy tolerance of the application scenario and is typically between 0.7 and 0.9. For example, if the threshold is set to 0.85, data pairs with a fuzzy similarity exceeding 0.85 are considered potentially redundant and added to the pre-screened redundant dataset. In practical applications, the pre-screening process may generate a large number of candidate redundant pairs. To improve efficiency, hashing or indexing techniques can be used to accelerate similarity calculations.

[0052] The local density value corresponding to the initially screened redundant data is calculated based on the distance between data points and the cutoff distance parameter, further filtering out redundant data. Local density reflects the distribution of data around a data point and is calculated by counting or weighting the sum of points whose distance from the point is less than the cutoff distance. The cutoff distance is a preset parameter that determines the range of local density calculations and can be determined through data analysis or cross-validation. For example, if the average neighbor distance of a data set is 0.1, the cutoff distance can be set to 0.15 to ensure that the local density calculation of each point includes a sufficient number of neighboring points.

[0053] The initially screened redundant data is refined based on the local density value, and data with a local density value greater than a preset density threshold is marked as candidate redundant data. The density threshold is set based on the overall density distribution of the dataset and can be the upper quartile or a specific percentile of the dataset's local density distribution. For example, if the local density values ​​of a dataset are distributed in the range [1, 10] with a median of 4, the density threshold can be set to 6, and data points with a local density greater than 6 can be marked as candidate redundant points. The purpose of density screening is to retain points located in high-density areas of the dataset. These points typically have high information redundancy and can be considered representative of that area.

[0054] Calculate a completeness score for candidate redundant data to assess its suitability for removal from the storage set. The completeness score reflects the completeness of the information contained in the data point and can be calculated based on information entropy, feature coverage, or data quality metrics. The calculation method for the completeness score varies depending on the data type: for text data, vocabulary coverage and semantic completeness can be considered; for image data, clarity and content richness can be considered; and for sensor data, sampling completeness and signal-to-noise ratio can be considered. Completeness scores are typically normalized to the range [0, 1], with higher values ​​indicating better data completeness.

[0055] Candidate redundant data with integrity scores below a pre-set integrity threshold is removed to ensure that data marked as redundant does not result in information loss. Setting the integrity threshold requires a balance between storage efficiency and information retention, typically between 0.3 and 0.7. For example, if the threshold is set to 0.5, candidate redundant data with integrity scores below 0.5 is retained (not removed), while candidate redundant data with integrity scores above 0.5 is confirmed as truly redundant data and can be removed from the storage set. The dataset to be stored is constructed based on the retained candidate redundant data. This dataset, after removing confirmed redundant data, is ultimately stored in the blockchain network.

[0056] In this embodiment, by mapping each feature dimension in the optimized feature sequence into a fuzzy membership value and dynamically adjusting the parameters of the triangular membership function based on the data distribution characteristics, an adaptive and flexible characterization of unstructured data features is achieved, thereby being able to more accurately reflect the fuzzy feature differences of different data. Data pairs with high similarity are identified through fuzzy similarity calculation, so that most redundant data can be effectively screened out in the early stage, thereby reducing the complexity of subsequent processing. The local density value is calculated using the distance between data points and the truncation distance parameter, and the initially screened redundant data is fine-screened based on the density threshold, so that data points with denser local distribution and stronger representativeness can be retained, thereby improving the accuracy of redundant data judgment. While maintaining data integrity, the amount of redundant data is effectively reduced, and high-quality construction of the data set is achieved, thereby improving data storage and management efficiency, and providing a more reliable data foundation for subsequent data analysis and modeling.

[0057] In an optional embodiment, Dividing the data set to be stored into blocks according to a preset size to obtain a data block set, calculating the access frequency of each data block in the data block set, executing a consistent hashing algorithm to allocate the data block set to obtain a data block storage location table, and establishing a data access index based on the data block storage location table includes: Preliminarily dividing the data set to be stored into blocks according to a preset size, calculating an optimal block factor including the total size of the data set and the data density variance, and adaptively dividing the data set to be stored into blocks according to the optimal block factor to obtain a data block set; For each data block in the data block set, calculating the access frequency of each data block in the data block set based on the number of historical accesses within a preset time window and a preset time decay factor; Calculating a storage node weight value based on the access frequency and a basic weight value corresponding to each storage node, performing virtual node expansion on the storage node based on the storage node weight value to obtain a virtual node distribution sequence, calculating a data block hash value for each data block in the data block set, and executing a consistent hashing algorithm to map a weighted sum of the data block hash value and the access frequency to a hash space corresponding to the virtual node distribution sequence to obtain a data block storage location table; Based on the data block storage location table, a main index table is established to record the mapping relationship between data blocks and physical nodes. Based on the access frequency, an access frequency threshold is set and a secondary index table for hot data access is established. The physical distance and logical distance are calculated, the routing weight is constructed and a routing table is generated. The main index table, the secondary index table and the routing table are combined to form a data access index.

[0058] The dataset to be stored is initially segmented into chunks based on a preset size. This size is typically a fixed value, such as 4MB or 8MB. During this initial segmentation process, continuous data is segmented into chunks based on the preset size. For text data, this can be done at natural paragraphs; for image data, it can be segmented at image boundaries; for audio data, it can be segmented at silent segments. After initial segmentation, a collection of similarly sized data chunks is formed. For example, if a 100MB dataset is initially segmented into chunks based on a preset size of 4MB, approximately 25 initial data chunks will be formed.

[0059] Calculate the optimal chunking factor, which takes into account the total dataset size and data density variance. The optimal chunking factor takes into account the total dataset size, node storage capacity, and data density distribution. Data density can be measured by information entropy; data blocks with higher information entropy contain more valid information and have higher density. The optimal chunking factor is calculated by multiplying the total dataset size by a normalization coefficient, then dividing the result by the sum of the data density variance and a tuning parameter. The normalization coefficient is related to the size of the blockchain network, and the tuning parameter is used to prevent the denominator from approaching zero. For example, for a dataset with a total size of 100MB and a data density variance of 0.05, the optimal chunking factor is approximately 166.7 when the normalization coefficient is 0.1 and the tuning parameter is 0.01.

[0060] The data set to be stored is adaptively partitioned according to the optimal partitioning factor to obtain a data block set. During the adaptive partitioning process, the results of the initial partitioning are adjusted so that areas with high information density are partitioned more finely and areas with low information density are partitioned more coarsely. In specific implementation, the information entropy of each initial data block is calculated and multiplied by the optimal partitioning factor to obtain the number of sub-blocks that the data block should be divided into. For example, if the information entropy of an initial data block is 0.8 and the optimal partitioning factor is 166.7, then the data block should be divided into approximately 133 sub-blocks. Considering the actual storage efficiency, a minimum sub-block size limit can be set, such as 10KB. After the adaptive partitioning is completed, the size of each data block in the formed data block set is inversely proportional to its information density, achieving optimized allocation of storage resources.

[0061] For each data block in the data block set, the access frequency is calculated based on the historical access count within a preset time window and a pre-set time decay factor. The preset time window can be one day, one week, or one month. The time decay factor is used to reduce the weight of older access records. The access frequency is calculated by multiplying the access count at each time point within the time window by the corresponding time decay value and then summing the results. The time decay value decreases as the time point increases from the current time, and an exponential decay function can be used. For example, if the number of accesses to a data block in the past seven days is [10, 5, 8, 12, 3, 0, 2], and the time decay factor is 0.8, then when calculating the access frequency, multiply the number of accesses per day by the corresponding decay value (1, 0.8, 0.64, 0.512, 0.41, 0.328, 0.262 from the most recent day to the most recent day, respectively), to obtain the weighted number of accesses [10, 4, 5.12, 6.14, 1.23, 0, 0.52]. The sum of the accesses yields an access frequency of approximately 27.01.

[0062] The storage node weight is calculated based on access frequency and the base weight of each storage node. The base weight is pre-set by the system administrator based on the actual situation of the node. The storage node weight is calculated by multiplying the node base weight by the weighted average of the access frequencies of all data blocks stored by the node. For example, if a storage node has a base weight of 1.2 and it stores three data blocks with access frequencies of 27.01, 15.6, and 8.4, the node weight is 1.2*(27.01+15.6+8.4) / 3=20.4.

[0063] Based on the storage node weights, virtual node expansion is performed on the storage nodes to generate a virtual node distribution sequence. Virtual node expansion creates multiple virtual nodes for each physical node, resulting in a more even data distribution. The number of virtual nodes is proportional to the node weight and can be calculated by multiplying the node weight by the system's preset virtual node base. For example, if the system's virtual node base is 100 and a node has a weight of 20.4, the node will be expanded to approximately 2040 virtual nodes. Virtual nodes are evenly distributed on the hash ring, with each virtual node corresponding to a hash value.

[0064] A data block hash value is calculated for each data block in the data block set. A hash algorithm such as SHA-256 or MD5 can be used. A consistent hashing algorithm is executed to map the weighted sum of the data block hash value and the access frequency to the hash space corresponding to the virtual node distribution sequence, resulting in a data block storage location table. The weighted sum is calculated by multiplying the data block hash value by the weight coefficient α, multiplying the access frequency by the weight coefficient (1-α), and then adding them. The weight coefficient α is usually set between 0.3 and 0.7 to balance the randomness of the hash value and the preference of the access frequency. The mapping process is to search the hash ring clockwise for the virtual node with the closest distance to the weighted sum. The physical node corresponding to this virtual node is the storage location of the data block. The data block storage location table contains the mapping relationship between data block IDs and storage node IDs, which is used for data storage and retrieval.

[0065] Based on the data block storage location table, a master index table is established to record the mapping relationship between data blocks and physical nodes. The master index table is a key-value pair structure, where the key is the data block ID and the value is the physical node ID where the data block is stored. The master index table supports fast queries with a query complexity of O(1) and is used to process storage location query requests for all data blocks. For example, the master index table may contain the following mapping: {data block 001: node A, data block 002: node B, data block 003: node A}, indicating that data blocks 001 and 003 are stored on node A, and data block 002 is stored on node B.

[0066] Based on the access frequency, an access frequency threshold is set and a secondary index table for hot data access is established. The access frequency threshold can be set as the average access frequency of all data blocks multiplied by a coefficient β, which is usually greater than 1, such as 1.5 or 2. Data blocks with access frequencies exceeding the threshold are marked as hot data and included in the secondary index table. The secondary index table contains a mapping between the hot data block ID and the storage node ID, as well as additional access frequency information. The secondary index table is used to optimize the access performance of hot data and can achieve cache preheating and load balancing.

[0067] Calculate physical and logical distances, construct routing weights, and generate a routing table. Physical distance refers to the network latency or geographic distance between nodes, while logical distance refers to the distance between nodes on the hash ring. Routing weights are calculated by taking the weighted average of the normalized physical and logical distances. Weights can be adjusted based on system requirements. The routing table contains information about the optimal path from the source node to the destination node and is used for routing decisions for data transmission. The primary index table, secondary index tables, and routing table form a data access index, providing comprehensive support for data storage and retrieval.

[0068] In this embodiment, by preliminarily dividing the stored data set into blocks according to a preset size, and calculating the optimal block factor based on the total size of the data set and the data density variance, and then performing adaptive block division based on the optimal block factor, the data block division can be made more balanced and reasonable, thereby avoiding waste of storage resources or decreased access efficiency due to uneven data distribution. For the block set of data blocks after block division, the access frequency is calculated based on the historical number of accesses within a preset time window and the time decay factor, which can dynamically reflect the actual usage popularity of the data block, thereby providing an accurate basis for subsequent storage optimization and access optimization. The storage node weight is calculated based on the access frequency and the basic weight value of the storage node, and virtual node expansion is performed, which can improve the balance of the consistent hash distribution, make the data blocks more reasonably distributed among the storage nodes, avoid data skew problems, and enhance the load balancing capability of the storage system, thereby achieving balanced data storage, improved access efficiency and overall system performance optimization, and providing reliable guarantees for efficient storage and fast access of large-scale data.

[0069] In an optional embodiment, Segment encoding is performed on each data block, and a hash value is calculated to obtain a data block feature table. Generating a data access verification sequence according to the data block feature table and the data access index includes: Obtain historical access records of data blocks and generate access pattern time series based on the historical access records; The content entropy, boundary sensitivity, and data density of the data block are calculated to construct a data block content feature vector. The optimal segment size is calculated based on the information entropy of the data block content feature vector and the variance of the access pattern time series. Each data block is segmented according to the optimal segment size to obtain a segment sequence. Obtain historical error records of data blocks and calculate the data block error rate; calculate the number of information symbols based on the access frequency of the data blocks in the segmented sequence; calculate the number of check symbols based on the data block error rate and the number of information symbols; encode the segmented sequence to obtain a coded block set; obtain a preset minimum segment size of the data block; calculate the tree structure depth based on the data block size and construct a tree structure; assign node numbers to the coded blocks in the coded block set to generate parent node pointers; calculate a hierarchical hash value for the coded block set; and form a data block feature table with the hierarchical hash value, tree structure depth information, and parent node pointers; The access frequency of the data block in the data access index is obtained to calculate the cache threshold, and the data nodes with access frequency greater than the cache threshold are cached and pre-fetched to establish a verification path. The current system time is obtained as the timestamp, and the read and write status, integrity status and access permission status of the data block are obtained to form the node status information. The hierarchical hash value, timestamp and node status information are signed to obtain a verification tag, and a data access verification sequence is generated based on the verification tag, verification path and data block feature table.

[0070] Obtain historical access records for data blocks. Historical access records include information such as access timestamps, access types (read / write), and access user IDs, and are typically stored in system logs. To generate access pattern time series based on historical access records, count the number of accesses within each time unit to form time series data. Time units can be set based on the application scenario, such as minutes, hours, or days. For example, the sequence of access counts for a data block within 24 consecutive hours might be [5, 3, 0, 1, 7, 12, 15, 20, 18, 10, 8, 6, 4, 2, 1, 0, 2, 5, 8, 10, 15, 12, 8, 6]. This sequence reflects the temporal distribution of data access, such as peak access times during work hours and low access times at night.

[0071] The content entropy, boundary sensitivity, and data density of a data block are calculated to construct a data block content feature vector. Content entropy reflects the randomness and information content of the data. It is calculated by summing the product of the probability of each symbol appearing in the statistical data and its logarithm, then taking the negative value. Boundary sensitivity measures the continuity of the data at segment boundaries. It is calculated by analyzing the degree of difference between adjacent data units. Data density indicates the amount of effective information contained in a unit of storage space. It is calculated by dividing the content entropy by the data block size. For example, for a data block containing a mixture of text and images, the resulting content feature vector might be [0.85, 0.32, 0.67], indicating a content entropy of 0.85, a boundary sensitivity of 0.32, and a data density of 0.67.

[0072] The optimal segment size is calculated based on the information entropy of the data block's content feature vector and the variance of the access pattern time series. Data with high information entropy contains more information and requires finer segmentation. Data with high access pattern variance has large fluctuations in access frequency and is therefore suitable for a variable-length segmentation strategy. The optimal segment size is calculated by multiplying the base segment size (e.g., 4KB) by a weighted function of information entropy and access variance. The weighting function can be designed as a weighted sum of the logarithm of information entropy and the square root of access variance, with the weights adjusted based on system performance requirements. For example, if a data block has an information entropy of 0.85, an access variance of 25, a base segment size of 4KB, and weights of 0.6 and 0.4, respectively, the optimal segment size is approximately 4KB × (0.6 × (-0.16) + 0.4 × 5) = 7.78KB.

[0073] Each data block is segmented according to the optimal segment size to produce a segment sequence. The segmentation process must consider the semantic boundaries of the data content. For text data, segmentation can be performed at paragraph or sentence boundaries; for image data, segmentation can be performed at image boundaries; for structured data, segmentation can be performed at record boundaries. The segmented sequence contains multiple data fragments with sizes close to the optimal segment size, each with location index information. For example, a 10MB data block with an optimal segment size of 8KB may produce approximately 1280 fragments after segmentation, each ranging in size from 7.5KB to 8.5KB.

[0074] Obtain the historical error records for the data block and calculate the data block error rate. Historical error records include information such as data read and write errors, verification failures, and node failures. The data block error rate is calculated by dividing the number of errors by the total number of accesses. For example, if a data block has a total of 10,000 accesses and 15 errors are recorded, the error rate is 0.0015. The number of information symbols is calculated based on the access frequency of the data block in the segmented sequence. The number of information symbols represents the number of symbols that actually contain information in the data and is related to the data size and encoding method. For binary data, the number of information symbols can be simplified to the number of bits; for text data, it can be the number of characters.

[0075] The number of check symbols is calculated based on the data block error rate and the number of information symbols. Check symbols are used for error detection and correction, and their number is related to the error rate and the number of information symbols. The calculation method is to multiply the number of information symbols by a function of the error rate. This function can be designed as the product of the logarithm of the error rate and an adjustment factor. For example, if the number of information symbols is 65536 (8KB of data), the error rate is 0.0015, and the adjustment factor is -2, the number of check symbols is approximately 65536 × (-2) × (-6.5) ≈ 852 symbols, accounting for approximately 1.3% of the total data volume. The segmented sequence is encoded to obtain a set of coded blocks. The encoding process uses forward error correction coding technology to combine information symbols and check symbols into coded blocks.

[0076] The preset minimum segment size of the data block is obtained. The tree structure depth is calculated based on the data block size and the tree structure is constructed. The tree structure depth is calculated by dividing the data block size by the minimum segment size, taking the logarithm, and then rounding up. For example, if the data block size is 10MB and the minimum segment size is 1KB, the tree structure depth is ceil(log(10×1024 / 1))=14. The constructed tree structure is a balanced tree, where each leaf node corresponds to a coding block, and non-leaf nodes contain the hash values ​​of child nodes. Node numbers are assigned to the coding blocks in the coding block set to generate parent node pointers. Node numbers can be assigned in a left-to-right and bottom-to-top order, and the parent node pointer points to the parent node number of the current node.

[0077] A hierarchical hash value is calculated for the set of coded blocks. This process begins at the leaf nodes and progresses upwards. The hash value of a leaf node is the hash value of the corresponding coded block's contents; the hash value of a non-leaf node is the concatenated hash value of all its child nodes. The hash algorithm uses a secure hash function. The hierarchical hash structure ensures that any change to a coded block will result in a change to the root node's hash value, facilitating data integrity verification. The hierarchical hash value, tree structure depth information, and parent node pointers are combined into a data block feature table. This uniquely identifies a data block on the blockchain and is used for data integrity verification and efficient retrieval.

[0078] The access frequency of data blocks in the data access index is obtained to calculate the cache threshold. The cache threshold is used to determine which data should be cached. The calculation method is to multiply the average access frequency of all data blocks by a coefficient factor. The coefficient factor can be dynamically adjusted based on system resource conditions and is usually between 1.2 and 2.0. For example, if the average access frequency of all data blocks in the system is 10 times / hour and the coefficient factor is 1.5, the cache threshold is 15 times / hour. Data nodes with access frequencies greater than the cache threshold are pre-fetched to establish a verification path. The verification path is a sequence of hash values ​​from leaf nodes to the root node.

[0079] The current system time is obtained as a timestamp. The timestamp format can be a Unix timestamp, accurate to milliseconds. The read / write status, integrity status, and access permission status of the data block are obtained to form the node status information. The read / write status indicates whether the data block is readable and writable; the integrity status indicates the result of the data integrity check; and the access permission status indicates the list of users or roles allowed to access. The hierarchical hash value, timestamp, and node status information are signed to obtain a verification tag. The signing process uses an asymmetric encryption algorithm, such as RSA or ECDSA, to ensure the unforgeability of the verification tag. A data access verification sequence is generated based on the verification tag, verification path, and data block feature table. The verification sequence contains all the necessary information to enable the client to verify the integrity, timeliness, and legitimacy of the data.

[0080] In this embodiment, by obtaining the historical access records of the data block and generating the access pattern time series, the access rules of the data block in different time periods can be extracted, thereby providing a dynamic reference for subsequent segmentation and cache optimization, calculating the content entropy, boundary sensitivity and data density of the data block content, constructing the data block content feature vector, and calculating the optimal segment size based on the information entropy of the content feature vector and the variance of the access pattern time series. Adaptive segmentation can be achieved according to the complexity of the data itself and the access volatility, making the data division more reasonable, which helps to improve data access efficiency and subsequent encoding performance. By obtaining the historical error records of the data block, the error rate is calculated. , and calculate the number of information symbols in combination with the access frequency of data blocks in the segmented sequence, and then determine the number of check symbols according to the error rate and the number of information symbols, and encode the segmented sequence. It can realize flexible check and encoding under different access intensity and error rate conditions, improve the reliability and error correction capability of data storage, calculate the cache threshold based on the data access index, and pre-cache the data nodes whose access frequency exceeds the threshold, which can effectively improve the access efficiency of high-frequency data and reduce access latency, and realize the coordinated improvement of data partition rationality, storage reliability, access efficiency and security credibility, providing efficient and reliable technical guarantee for large-scale data storage and access.

[0081] Figure 2 This is a flowchart of data block verification and processing of an unstructured data storage optimization method based on information entropy and blockchain in an embodiment of the present invention.

[0082] In an optional embodiment, Performing data integrity verification according to the data access verification sequence, generating a certificate containing data identification and signature, and recording the verification results to the blockchain includes: Obtaining a data access verification sequence, recursively verifying the hierarchical hash value based on the verification path in the data access verification sequence, and generating a verification result; Generate a data identifier based on the verification result, sign the data identifier to obtain an integrity signature, and combine the data identifier and the integrity signature to form a verification certificate; Submit the verification proof to the blockchain network and record it.

[0083] The data access verification sequence includes key information such as a verification mark, a verification path, and a data block feature table. The verification mark is generated by signing the hierarchical hash value, timestamp, and node state information, and is used to prove the authenticity and integrity of the data. The verification path is a sequence of hash values from the leaf node to the root node, and is used to verify whether a specific data block belongs to the original data set. The data block feature table includes the hierarchical hash value, tree structure depth information, and parent node pointer, and is used to locate and identify the data block. In actual application, the verification sequence can be obtained from the data provider through a secure channel, or obtained by querying the blockchain. For example, a certain data access verification sequence may include a verification mark "0x7a3bc4d...", a verification path ["0x1a2b3c...", "0x4d5e6f...", "0x7g8h9i..."], and a data block feature table {hash value: "0x9j8k7l...", depth: 5, parent node: [123, 456, 789]}.

[0084] The hierarchical hash value is recursively verified based on the verification path in the data access verification sequence to generate a verification result. The recursive verification process starts from the leaf node and verifies layer by layer upwards to the root node. The specific verification steps are as follows: take the first hash value in the verification path and compare it with the hash value of the current data block to confirm whether they are consistent; if they are consistent, combine the hash value with the next hash value in the verification path to calculate a new hash value; repeat the above process until the root node hash value is verified. If the final calculated root node hash value is consistent with the root hash value provided in the verification sequence, the verification is passed; otherwise, the verification fails. The hash algorithm used in the verification process should be consistent with the algorithm used to generate the hierarchical hash value, such as SHA-256. For example, assuming that the leaf node hash value is "0x1a2b3c...", and the verification path is ["0x4d5e6f...", "0x7g8h9i..."], the verification process is: calculate Hash("0x1a2b3c..."+"0x4d5e6f...") to get the intermediate hash value "0xabcd12..."; then calculate Hash("0xabcd12..."+"0x7g8h9i...") to get the root hash value "0x9876fe..."; finally, compare "0x9876fe..." with the expected root hash value to determine whether the verification is passed.

[0085] Verification results include information such as the verification status (pass / fail), verification time, and verification node ID. The verification status directly reflects the integrity and authenticity of the data, the verification time records the execution time of the verification operation, and the verification node ID indicates the blockchain node that performed the verification. Verification results can be expressed in JSON format for easy storage and transmission, such as {"status":"verified","timestamp":1629456789","node_id":"node012345","details":{"hash_match":true","time_valid":true}}.

[0086] Based on the verification results, a data identifier is generated. This unique identifier serves as a reference for the data on the blockchain and is used for subsequent reference and retrieval. This identifier is generated by concatenating key fields from the verification result (such as the verification status, timestamp, and data hash value) and calculating a hash value. The hash algorithm uses a secure hash function such as SHA-256 to ensure uniqueness and unforgeability. For example, if the verification result contains the verification status "verified," the timestamp 1629456789, and the data hash value "0x9876fe...", these values ​​can be concatenated and hashed to produce the data identifier "0xa1b2c3d4...".

[0087] The data identifier is signed to generate an integrity signature. The signing process encrypts the data identifier with a private key, generating an unforgeable digital signature. The signature algorithm can utilize asymmetric encryption algorithms such as RSA and ECDSA to ensure the security and verifiability of the signature. The signature result is a fixed-length string, such as "0xe5f6g7h8...". The signature includes information such as the signature algorithm identifier, signature value, and signing time, ensuring signature integrity and traceability. An integrity signature can be represented as {"algorithm":"ECDSA","value":"0xe5f6g7h8..."","timestamp":1629456790}.

[0088] The data identifier and integrity signature form a verification proof, which is used to prove the data's credibility to a third party. The verification proof structure includes fields such as the data identifier, integrity signature, proof generation time, and proof version number, forming a complete proof document. The verification proof can be serialized in JSON or XML format for easy storage and transmission. For example, a verification proof can be represented as {"data_id":"0xa1b2c3d4...","signature":{"algorithm":"ECDSA","value":"0xe5f6g7h8...","timestamp":1629456790},"proof_time":1629456795,"version":"1.0"}}.

[0089] The verification proof is submitted to the blockchain network and recorded. The submission process includes constructing a blockchain transaction, including the verification proof as transaction data; signing the transaction with the sender's private key; broadcasting the signed transaction to the blockchain network; and waiting for the transaction to be included in a block and confirmed. After receiving the transaction, the blockchain network verifies the validity of the transaction signature and adds the valid transaction to a candidate block. Through a consensus mechanism (such as proof-of-work or proof-of-stake), the candidate block is confirmed as a new block and added to the blockchain. Once the transaction is confirmed, the verification proof is permanently recorded on the blockchain and can be retrieved and verified by anyone.

[0090] In this embodiment, by obtaining a data access verification sequence and recursively verifying the hierarchical hash value based on the verification path, the integrity of the data during the storage and access process can be checked step by step, thereby effectively discovering and preventing data tampering or loss. A data identifier is generated based on the verification result and signature processing is performed to obtain an integrity signature, which can ensure that the source of the data is credible and the content is authentic during transmission and storage, avoiding illegal forgery. The data identifier and integrity signature are combined into a verification certificate and submitted to the blockchain network for recording. The immutability and traceability of the blockchain can be utilized to realize decentralized evidence storage of data access and integrity verification, thereby greatly improving the data security and credibility of the system, realizing the automation and tamper-proof protection of data integrity verification, and ensuring the reliability and traceability of data in a distributed environment.

[0091] A second aspect of an embodiment of the present invention provides an unstructured data storage optimization system based on information entropy and blockchain, including: The first unit is used to establish a blockchain storage network and obtain unstructured data, construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain dimensionality-reduced data; The second unit is configured to map the reduced-dimensional data to a high-dimensional feature space using a preset kernel function, construct an optimal classification hyperplane, calculate a distance vector from each reduced-dimensional data to the optimal classification hyperplane to obtain an initial redundant data set, generate a binary code sequence based on data features in the initial redundant data set, perform selection, crossover, and mutation operations on the binary code sequence to obtain an optimized feature sequence, label redundant data according to the optimized feature sequence, calculate fuzzy similarity of different redundant data, and eliminate data whose fuzzy similarity is greater than a preset fuzzy similarity threshold to obtain a data set to be stored; A third unit is configured to divide the data set to be stored into blocks according to a preset size to obtain a data block set, calculate the access frequency of each data block in the data block set, execute a consistent hashing algorithm to allocate the data block set based on the access frequency to obtain a data block storage location table, establish a data access index based on the data block storage location table, perform segmented encoding on each data block and calculate a hash value to obtain a data block feature table, and generate a data access verification sequence based on the data block feature table and the data access index; The fourth unit is used to perform data integrity verification according to the data access verification sequence, generate a certificate including the data identification and signature, and record the verification result to the blockchain.

[0092] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0093] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0094] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unstructured data storage optimization method based on information entropy and blockchain, characterized in that: include: Establish a blockchain storage network and obtain unstructured data, construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain dimensionality-reduced data; The reduced-dimensional data is mapped to a high-dimensional feature space using a preset kernel function, an optimal classification hyperplane is constructed, a distance vector from each reduced-dimensional data to the optimal classification hyperplane is calculated to obtain an initial redundant data set, a binary coding sequence is generated based on the data features in the initial redundant data set, selection, crossover, and mutation operations are performed on the binary coding sequence to obtain an optimized feature sequence, redundant data is labeled according to the optimized feature sequence, fuzzy similarity of different redundant data is calculated, and data with a fuzzy similarity greater than a preset fuzzy similarity threshold is eliminated to obtain a data set to be stored; Dividing the data set to be stored into blocks according to a preset size to obtain a data block set, calculating the access frequency of each data block in the data block set, executing a consistent hashing algorithm based on the access frequency to allocate the data block set to obtain a data block storage location table, establishing a data access index based on the data block storage location table, performing segmented encoding on each data block and calculating a hash value to obtain a data block feature table, and generating a data access verification sequence based on the data block feature table and the data access index; Perform data integrity verification according to the data access verification sequence, generate a certificate containing data identification and signature, and record the verification result to the blockchain.

2. The method according to claim 1, characterized in that Establish a blockchain storage network and obtain unstructured data. Construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain the reduced dimensionality data including: Obtain unstructured data and establish a blockchain storage network consisting of multiple nodes; Extracting frequency features and time series features from the unstructured data, calculating feature vectors based on the frequency features and the time series features, and constructing a covariance matrix based on the feature vectors; Performing eigenvalue decomposition on the covariance matrix to obtain initial eigenvalues, reconstructing the unstructured data based on the initial eigenvalues ​​to obtain reduced-dimensionality data, and storing the reduced-dimensionality data in the blockchain storage network.

3. The method according to claim 1, characterized in that The reduced-dimensional data are mapped to a high-dimensional feature space using a preset kernel function, an optimal classification hyperplane is constructed, a distance vector from each reduced-dimensional data to the optimal classification hyperplane is calculated to obtain an initial redundant data set, a binary code sequence is generated based on the data features in the initial redundant data set, and selection, crossover, and mutation operations are performed on the binary code sequence to obtain an optimized feature sequence, including: Obtaining dimensionality-reduced data, calculating the Euclidean distance between different data points in the dimensionality-reduced data, constructing a Gaussian kernel function based on the Euclidean distance and a preset kernel function bandwidth parameter, and mapping the dimensionality-reduced data to a high-dimensional feature space using the Gaussian kernel function to obtain high-dimensional feature data; Constructing an optimization objective function based on the high-dimensional feature data, wherein the optimization objective function includes a normal vector modulus minimization term and a constraint relaxation term weighted by a preset penalty factor, and solving the optimization objective function to obtain a normal vector and a bias term of an optimal classification hyperplane; Calculating a distance value from each data point in the dimensionality reduction data to the optimal classification hyperplane based on the normal vector and the bias term, sorting the distance values ​​from small to large, selecting the data points in the top 10% to construct an initial redundant data set, extracting a feature vector corresponding to each data point from the initial redundant data set, calculating the mean of feature components of each dimension of the feature vector, comparing the feature components with the mean of the corresponding dimension, assigning a binary value of 1 to feature components greater than the mean, and assigning a binary value of 0 to feature components less than the mean, to generate a binary code sequence; The fitness value of each sequence in the binary code sequence is calculated, a roulette wheel selection operation is performed based on the fitness value to construct a high-quality sequence, a single-point crossover operation is performed on the high-quality sequence to generate a crossover sequence, and a random inversion operation is performed on the sites of the crossover sequence to obtain an optimized feature sequence.

4. The method according to claim 1, wherein Redundant data are marked according to the optimized feature sequence, fuzzy similarities of different redundant data are calculated, and data with fuzzy similarities greater than a preset fuzzy similarity threshold are eliminated to obtain a data set to be stored, including: Obtaining an optimized feature sequence and mapping each feature dimension in the optimized feature sequence into a fuzzy membership value, constructing a triangular membership function based on the fuzzy membership value, and dynamically adjusting parameters of the triangular membership function based on data distribution characteristics corresponding to the unstructured data to obtain an adaptive fuzzy feature; Building a multidimensional fuzzy rule base based on the adaptive fuzzy feature, assigning an initial weight coefficient to each rule in the multidimensional fuzzy rule base, calculating the fuzzy similarity between different data according to the multidimensional fuzzy rule base, and marking data pairs with fuzzy similarity greater than a preset fuzzy similarity threshold as pre-screened redundant data; Calculating the local density value corresponding to the primary screening redundant data based on the distance between data points and the cutoff distance parameter, fine-screening the primary screening redundant data according to the local density value, and marking the data with a local density value greater than a preset density threshold as candidate redundant data; The integrity scores of the candidate redundant data are calculated, candidate redundant data with integrity scores lower than a preset integrity threshold are eliminated, and a data set to be stored is constructed based on the retained candidate redundant data.

5. The method according to claim 1, wherein Dividing the data set to be stored into blocks according to a preset size to obtain a data block set, calculating the access frequency of each data block in the data block set, executing a consistent hashing algorithm to allocate the data block set to obtain a data block storage location table, and establishing a data access index based on the data block storage location table includes: Preliminarily dividing the data set to be stored into blocks according to a preset size, calculating an optimal block factor including the total size of the data set and the data density variance, and adaptively dividing the data set to be stored into blocks according to the optimal block factor to obtain a data block set; For each data block in the data block set, calculating the access frequency of each data block in the data block set based on the number of historical accesses within a preset time window and a preset time decay factor; Calculating a storage node weight value based on the access frequency and a basic weight value corresponding to each storage node, performing virtual node expansion on the storage node based on the storage node weight value to obtain a virtual node distribution sequence, calculating a data block hash value for each data block in the data block set, and executing a consistent hashing algorithm to map a weighted sum of the data block hash value and the access frequency to a hash space corresponding to the virtual node distribution sequence to obtain a data block storage location table; Based on the data block storage location table, a main index table is established to record the mapping relationship between data blocks and physical nodes. Based on the access frequency, an access frequency threshold is set and a secondary index table for hot data access is established. The physical distance and logical distance are calculated, the routing weight is constructed and a routing table is generated. The main index table, the secondary index table and the routing table are combined to form a data access index.

6. The method according to claim 1, characterized in that Segment encoding is performed on each data block, and a hash value is calculated to obtain a data block feature table. Generating a data access verification sequence according to the data block feature table and the data access index includes: Obtain historical access records of data blocks and generate access pattern time series based on the historical access records; The content entropy, boundary sensitivity, and data density of the data block are calculated to construct a data block content feature vector. The optimal segment size is calculated based on the information entropy of the data block content feature vector and the variance of the access pattern time series. Each data block is segmented according to the optimal segment size to obtain a segment sequence. Obtain historical error records of data blocks and calculate the data block error rate; calculate the number of information symbols based on the access frequency of the data blocks in the segmented sequence; calculate the number of check symbols based on the data block error rate and the number of information symbols; encode the segmented sequence to obtain a coded block set; obtain a preset minimum segment size of the data block; calculate the tree structure depth based on the data block size and construct a tree structure; assign node numbers to the coded blocks in the coded block set to generate parent node pointers; calculate a hierarchical hash value for the coded block set; and form a data block feature table with the hierarchical hash value, tree structure depth information, and parent node pointers; The access frequency of the data block in the data access index is obtained to calculate the cache threshold, and the data nodes with access frequency greater than the cache threshold are cached and pre-fetched to establish a verification path. The current system time is obtained as the timestamp, and the read and write status, integrity status and access permission status of the data block are obtained to form the node status information. The hierarchical hash value, timestamp and node status information are signed to obtain a verification tag, and a data access verification sequence is generated based on the verification tag, verification path and data block feature table.

7. The method according to claim 1, characterized in that Performing data integrity verification according to the data access verification sequence, generating a certificate containing data identification and signature, and recording the verification results to the blockchain includes: Obtaining a data access verification sequence, recursively verifying the hierarchical hash value based on the verification path in the data access verification sequence, and generating a verification result; Generate a data identifier based on the verification result, sign the data identifier to obtain an integrity signature, and combine the data identifier and the integrity signature to form a verification certificate; Submit the verification proof to the blockchain network and record it.

8. An unstructured data storage optimization system based on information entropy and blockchain, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to establish a blockchain storage network and obtain unstructured data, construct the covariance matrix corresponding to the unstructured data and perform eigenvalue decomposition to obtain dimensionality-reduced data; The second unit is configured to map the reduced-dimensional data to a high-dimensional feature space using a preset kernel function, construct an optimal classification hyperplane, calculate a distance vector from each reduced-dimensional data to the optimal classification hyperplane to obtain an initial redundant data set, generate a binary code sequence based on data features in the initial redundant data set, perform selection, crossover, and mutation operations on the binary code sequence to obtain an optimized feature sequence, label redundant data according to the optimized feature sequence, calculate fuzzy similarity of different redundant data, and eliminate data whose fuzzy similarity is greater than a preset fuzzy similarity threshold to obtain a data set to be stored; A third unit is configured to divide the data set to be stored into blocks according to a preset size to obtain a data block set, calculate the access frequency of each data block in the data block set, execute a consistent hashing algorithm to allocate the data block set based on the access frequency to obtain a data block storage location table, establish a data access index based on the data block storage location table, perform segmented encoding on each data block and calculate a hash value to obtain a data block feature table, and generate a data access verification sequence based on the data block feature table and the data access index; The fourth unit is used to perform data integrity verification according to the data access verification sequence, generate a certificate including the data identification and signature, and record the verification result to the blockchain.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images