Data storage method and system based on data compression technology

Through data compression technology, similar data blocks are identified and spliced ​​to form a storage matrix of feature representation, which solves the problems of redundant data and complex analysis in traditional data storage, realizes efficient data storage and rapid retrieval, and improves data processing efficiency and cross-border Domain analysis capability.

CN118915976AInactive Publication Date: 2024-11-08杭州嘉识科技有限公司

Patent Information

Application Number
CN202411411587.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-11
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In traditional data storage methods, redundant data leads to waste of storage space and complex data analysis, especially in multi-dimensional data analysis, which leads to long retrieval and analysis time and large computing resources consumption.

Method used

Through data compression technology, similar but incompletely repeated data blocks are identified and spliced, converted into feature representations, forming a storage matrix, and defining a recovery algorithm based on metadata index to obtain the original data blocks.

Benefits of technology

It significantly reduces the need for redundant data storage, improves data access speed and processing efficiency, provides a standardized input format, and promotes cross-domain data analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915976A_ABST
    Figure CN118915976A_ABST
Patent Text Reader

Abstract

The invention is suitable for the technical field of data storage, and provides a data storage method and system based on a data compression technology, and the system comprises a data reading and similarity analysis module, a data type reading and splicing module, a storage matrix generation module, and a metadata indexing and recovery module. By identifying and splicing similar data blocks, storage of redundant data is reduced. According to the method, the storage requirement can be remarkably reduced, and especially when a large-scale data set is processed, the occupied space of a database is remarkably reduced. The method is particularly important for the environment with limited resources. Similar data blocks are merged and converted into feature representations, and a formed storage matrix enables data to be more orderly and structured. According to the structured storage mode, the data access speed can be increased, and particularly, required information can be quickly positioned during retrieval and analysis, so that the overall data processing efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data storage technology, and in particular to a data storage method and system based on data compression technology. Background Art

[0002] The field of data storage technology is an important part of modern information technology, involving a variety of methods and devices for storing and managing data. Data compression technology can not only reduce the storage space occupied, but also improve the data transmission efficiency and cost-effectiveness.

[0003] In traditional data storage methods, the existence of redundant data often leads to a waste of storage space, and the data storage structure is often relatively scattered, resulting in more time and computing resources required for retrieval and analysis. The storage format may make the data analysis process complicated and time-consuming, especially in multidimensional data analysis. Summary of the invention

[0004] The purpose of the present invention is to provide a data storage method and system based on data compression technology, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0005] The present invention is implemented in this way: a data storage method based on data compression technology, the method comprising: Read the data content, perform similarity analysis on the data, set a similarity threshold, identify and obtain similar but not completely duplicated data blocks in the data; Read the acquired data type, select a splicing rule according to the data type, splice the identified similar data blocks, obtain a spliced ​​merged block, and convert the merged block into a feature representation; Define a storage matrix, where the rows of the matrix represent data blocks and the columns represent features, and fill the data into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and extract the final storage matrix; Create a metadata index, define a recovery algorithm based on the acquired storage matrix, and decode according to the stored metadata and splicing logic to obtain the original data block.

[0006] As a further solution of the present invention, the similarity analysis of the data, setting the similarity threshold, identifying and obtaining similar but not completely duplicated similar data blocks in the data specifically includes: Read the data set and identify the data format; Identify duplicate data in the data set, mark these duplicates with indexes to obtain several data blocks, and count the number of repetitions and related features of each data block; Based on the relevant features of each data block, construct the feature vector of each data block, calculate the similarity of all data blocks pairwise, and generate a similarity matrix; A similarity threshold is set, and the similarity matrix is ​​screened based on the similarity threshold, similar data blocks that meet the similarity threshold are identified, and a similar data set list is generated, wherein the similar data set list only includes similar data blocks that meet the similarity threshold.

[0007] As a further solution of the present invention, the method of selecting a splicing rule according to the data type, splicing the identified similar data blocks to obtain a spliced ​​merged block, and converting the merged block into a feature representation specifically includes: Classify the identified similar data blocks, define splicing rules for each category, and perform actual splicing operations on the identified similar data blocks according to the selected rules to obtain the spliced ​​merged blocks; Extract key features from the merged blocks and convert the extracted features into vector format; Evaluate the structure and logic of the spliced ​​data and compare their similarity with the original data.

[0008] As a further solution of the present invention, the storage matrix is ​​defined, wherein the rows of the matrix represent data blocks and the columns represent features, and the data is filled into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and the final storage matrix is ​​extracted, which specifically includes: Define the storage matrix structure, determine the number of rows as the number of merged data blocks, and each row corresponds to a data block; determine the number of columns as the number of features, and each column corresponds to a feature; Create an empty storage matrix according to predefined dimensions ; Traverse the merged data block features one by one ,in, Indicates samples, Indicates Features, find the corresponding feature column for each data block, and then fill the data block feature value to the corresponding position of the storage matrix to accurately map the extracted feature data to the specified position of the storage matrix to obtain the filled storage matrix: ; Read all filled matrix positions in the storage matrix, and fill the unfilled positions with default values ​​to form the final storage matrix.

[0009] As a further solution of the present invention, the step of creating a metadata index, defining a recovery algorithm based on the acquired storage matrix, and decoding according to the stored metadata and splicing logic to obtain the original data block specifically includes: For each row, extract the location and specific values ​​of all corresponding eigenvalues; According to the splicing logic extracted from the metadata index, the feature values ​​of the row after feature extraction are sorted and spliced ​​to reconstruct the original data block; The concatenated result is stored in the previously initialized storage space, corresponding one-to-one to the original identifier or index of the data block; This data is compared upon recovery to verify the consistency of the decoded entire block with the original data.

[0010] Another object of the present invention is to provide a data storage system based on data compression technology, the system comprising: The data reading and similarity analysis module is used to read the data content, perform similarity analysis on the data, set the similarity threshold, and identify and obtain similar but not completely repeated similar data blocks in the data; The data type reading and splicing module is used to read the acquired data type, select the splicing rule according to the data type, splice the identified similar data blocks, obtain the spliced ​​merged block, and convert the merged block into a feature representation; The storage matrix generation module is used to define a storage matrix, where the matrix rows represent data blocks and the columns represent features, and fill the data into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and extract the final storage matrix; The metadata index and recovery module is used to create a metadata index, define a recovery algorithm based on the acquired storage matrix, and decode according to the stored metadata and splicing logic to obtain the original data block.

[0011] As a further solution of the present invention, the data reading and similarity analysis module includes: A data reading unit, used for reading a data set and identifying a data format; A repeated data identification unit is used to identify repeated data in a data set, mark these repeated items with indexes to obtain several data blocks, and count the number of repetitions and related features of each data block; A feature vector construction unit is used to construct a feature vector of each data block based on relevant features of each data block, calculate similarities between all data blocks, and generate a similarity matrix; The similarity screening unit is used to set a similarity threshold, screen the similarity matrix based on the similarity threshold, identify similar data blocks that meet the similarity threshold, and generate a similar data set list, wherein the similar data set list only includes similar data blocks that meet the similarity threshold.

[0012] As a further solution of the present invention, the data type reading and splicing module includes: A data block classification unit is used to classify the identified similar data blocks, define a splicing rule for each category, and perform an actual splicing operation on the identified similar data blocks according to the selected rule to obtain a merged block after splicing; A feature extraction unit is used to extract key features from the merged blocks after splicing, and uniformly convert the extracted features into vector format; The similarity evaluation unit is used to evaluate the structure and logic after splicing and compare its similarity with the original data.

[0013] As a further solution of the present invention, the storage matrix generation module includes: The matrix structure definition unit is used to define the storage matrix structure, determine the number of rows as the number of merged data blocks, and each row corresponds to a data block; determine the number of columns as the number of features, and each column corresponds to a feature; Empty matrix creation unit, used to create an empty storage matrix according to predefined dimensions ; Matrix filling unit, used to traverse the merged data block features one by one ,in, Indicates samples, Indicates Features, find the corresponding feature column for each data block, and then fill the data block feature value to the corresponding position of the storage matrix to accurately map the extracted feature data to the specified position of the storage matrix to obtain the filled storage matrix: ; The matrix completion unit is used to read all filled matrix positions in the storage matrix, and fill the unfilled positions with default values ​​to form a final storage matrix.

[0014] As a further solution of the present invention, the metadata indexing and recovery module includes: A feature content extraction unit, used to extract the positions and specific values ​​of all corresponding feature values ​​for each row; A splicing logic reconstruction unit, used for sorting and splicing the feature values ​​of the row after feature extraction according to the splicing logic extracted from the metadata index, and reconstructing the original data block; A result storage unit, used to store the concatenated result in a previously initialized storage space, in one-to-one correspondence with the original identifier or index of the data block; The consistency verification unit is used to compare this data during recovery and verify the consistency of the decoded entire block with the original data.

[0015] The beneficial effects of the present invention are: By identifying and concatenating similar data blocks, redundant data storage is reduced. This approach can significantly reduce storage requirements, especially when dealing with large data sets, significantly reducing the space occupied by the database. This is particularly important in resource-constrained environments. By merging similar data blocks and converting them into feature representations, the resulting storage matrix makes the data more organized and structured. This structured storage method can speed up data access, especially when searching and analyzing, it can quickly locate the required information, thereby improving overall data processing efficiency.

[0016] The featurized representation of the storage matrix provides a standardized input format for machine learning and data mining. This standardization not only improves the speed and accuracy of model training, but also makes it easier to integrate data with other data sets, facilitating cross-domain data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a data storage method based on data compression technology provided by an embodiment of the present invention; Figure 2 A flowchart of performing similarity analysis on data, setting a similarity threshold, and identifying and obtaining similar but not completely duplicated similar data blocks in the data provided by an embodiment of the present invention; Figure 3 A flowchart of selecting a splicing rule according to data type and splicing identified similar data blocks provided by an embodiment of the present invention; Figure 4 A flow chart of defining a storage matrix provided by an embodiment of the present invention, wherein the rows of the matrix represent data blocks, the columns represent features, and filling data into corresponding positions of the storage matrix according to the feature representation after the merged blocks are transformed; Figure 5 A flowchart of an embodiment of the present invention for defining a recovery algorithm based on an acquired storage matrix and performing decoding according to stored metadata and splicing logic; Figure 6 A structural block diagram of a data storage system based on data compression technology provided by an embodiment of the present invention; Figure 7 A structural block diagram of a data reading and similarity analysis module provided by an embodiment of the present invention; Figure 8A structural block diagram of a data type reading and splicing module provided in an embodiment of the present invention; Fig. 9 A structural block diagram of a storage matrix generation module provided in an embodiment of the present invention; Fig.10 A structural block diagram of a metadata indexing and recovery module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0020] Figure 1 A flowchart of a data storage method based on data compression technology provided by an embodiment of the present invention, such as Figure 1 As shown, a data storage method based on data compression technology, the method includes: S100, reading data content, performing similarity analysis on the data, setting a similarity threshold, identifying and obtaining similar data blocks that are similar but not completely repeated in the data; This step will obtain the necessary data from the storage medium and adjust the subsequent processing strategy according to the data type, such as text, image, audio, video, etc. In addition, this step also includes identifying duplicate data in the data set and marking these duplicates with indexes to generate an initial set of data blocks. To achieve this goal, each data block is hashed using a hash algorithm to generate a unique hash value. If the hash value is repeated, it is marked as duplicate data. This processing ensures that there is a preliminary understanding of the basic organization and distribution of the data set.

[0021] The next step is to count the number of repetitions and related features of each data block. In order to process the data more accurately in the subsequent processing, it is particularly important to create a huge data block feature database. The feature vectors of each data block are recorded, and these features include but are not limited to length, content features, structural features, etc. These feature vectors will become the basis for subsequent similarity calculations. In addition, the process of constructing the feature vectors of each data block and generating a similarity matrix can further optimize this step. The application of feature extraction algorithms is also a crucial link. For example, the TF-IDF algorithm can be used for text data, and SIFT or SURF can be used for image data to generate feature vectors. In addition, in order to measure the similarity of data blocks, the similarity calculation can use cosine similarity or Jaccard similarity coefficient, and all data blocks are calculated pairwise to generate a similarity matrix.

[0022] Finally, set the similarity threshold and filter similar data blocks. This step first requires combining actual business needs or determining the appropriate similarity threshold through experiments, then filtering out similar data blocks with similarity higher than the threshold, and forming a similar data set list with these data blocks for subsequent splicing and storage optimization. The processing here not only optimizes the data storage strategy, but also lays a foundation for unlimited imagination for subsequent processing.

[0023] This step demonstrates great flexibility. Its feature extraction and similarity calculation methods can be changed and adjusted for different types of data sets, improving the overall adaptability and general performance of the system. When processing big data and diverse data types, this flexibility can bring unparalleled operational advantages, thereby meeting the storage and management needs in diversified scenarios. Overall, these significant advantages obviously improve the whole process without tedious and complicated steps. It is extremely efficient and easy to execute, providing excellent support and optimization effects for the overall data compression technology.

[0024] like Figure 2 As shown, the similarity analysis of the data, setting the similarity threshold, identifying and obtaining similar but not completely repeated similar data blocks in the data, specifically includes: S110, reading a data set and identifying a data format; S120, identifying repeated data in the data set, marking the repeated items with indexes to obtain a number of data blocks, and counting the number of repetitions and related features of each data block; S130, constructing a feature vector of each data block based on relevant features of each data block, calculating similarities between all data blocks, and generating a similarity matrix; S140, setting a similarity threshold, and screening the similarity matrix based on the similarity threshold, identifying similar data blocks that meet the similarity threshold, and generating a similar data set list, wherein the similar data set list only includes similar data blocks that meet the similarity threshold.

[0025] S200, reading the acquired data type, selecting a splicing rule according to the data type, splicing the identified similar data blocks to obtain a spliced ​​merged block, and converting the merged block into a feature representation; In this step, data blocks are clustered according to data content, format and characteristics, and applicable splicing rules are defined for each type of data block. The specific method is to classify data blocks through supervised learning or unsupervised learning algorithms, and run a specific splicing process through selected data blocks to start the actual splicing operation. Based on the selected splicing rules, the identified similar data blocks are spliced ​​to form a spliced ​​merged block.

[0026] After the splicing is completed, it is necessary to extract key features from these spliced ​​merged blocks and convert all features into a standardized vector format. To achieve this goal, the core features of the data are first obtained through different extraction algorithms. For example, the word embedding model is used for text data, and the convolutional neural network is used to extract features for image data. Then, these features are uniformly converted into vector representations to ensure the computational nominal driving effect in the next step of the evaluation process.

[0027] Finally, the structure and logic of the spliced ​​data are evaluated, which is an important feedback adjustment link. By calculating the similarity of the spliced ​​data, it can be compared with the original data for evaluation. For specific types of similarity algorithms, that is, cosine similarity and edit distance can be used for text, and PSNR, SSIM, etc. can be used for images to ensure the accuracy and completeness of the spliced ​​merged blocks at the logical or semantic level, evaluate the results, and continuously improve the splicing strategy.

[0028] By classifying similar data blocks and defining dedicated rules for each category, highly customized data splicing rule settings can be achieved. This ensures that operational efficiency can be highly optimized in a strict environment, and reliable and flexible splicing processing of various types of data can be guaranteed in unpredictable business scenarios.

[0029] By merging similar data blocks, not only data redundancy is significantly reduced, but also the efficiency of storage and subsequent access is improved. After the spliced ​​merged blocks are extracted and converted into vector representation, the data storage format is more standardized, which is fully prepared for the subsequent defined storage matrix to fill the data. Customized feature extraction and unified vectorization ensure consistency, provide raw materials for subsequent evaluation links, and make the management and indexing of the final data simple and scientific.

[0030] like Figure 3 As shown, the method selects a splicing rule according to the data type, splices the identified similar data blocks to obtain a spliced ​​merged block, and converts the merged block into a feature representation, specifically including: S210, classifying the identified similar data blocks, defining a splicing rule for each category, and performing an actual splicing operation on the identified similar data blocks according to the selected rule to obtain a spliced ​​merged block; S220, extracting key features from the merged blocks after splicing, and uniformly converting the extracted features into a vector format; S230, evaluating the structure and logic after splicing and comparing the similarity with the original data.

[0031] S300, defining a storage matrix, wherein the rows of the matrix represent data blocks and the columns represent features, and filling the data into corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and extracting the final storage matrix; In this step, we first need to determine the number of rows of the storage matrix according to the number of merged data blocks. Each row represents a merged data block to ensure that the features of each data block can be accurately tracked. At the same time, we set the number of columns according to the number of features, and each column corresponds to a specific feature. On this basis, we create an empty storage matrix with initial values ​​usually set to zero or other default values ​​for subsequent filling.

[0032] Next, the features of the merged data blocks are traversed one by one. For each data block’s feature value, its corresponding column index needs to be found. Usually, a feature mapping table can be established to quickly find the corresponding column. After finding the corresponding feature column, the feature value of the data block is filled into the corresponding position of the storage matrix to ensure accurate mapping of the feature value of each data block, thereby avoiding errors in data storage.

[0033] After completing the feature filling of all merged blocks, read all the filled matrix positions in the storage matrix, and fill the unfilled positions with default values ​​(such as zero or other specific values) to form the final storage matrix. This process ensures the integrity of the matrix and avoids the null value problem in subsequent data processing.

[0034] By concentrating the features of similar data blocks in a matrix, efficient organization of data is achieved, which is convenient for subsequent retrieval and analysis. At the same time, merging similar data blocks avoids redundant storage, greatly reduces the demand for storage space, and improves storage efficiency. The flexibility of the storage matrix structure makes it easy to add new features as needed without significantly modifying the storage structure, thus facilitating future system expansion. In addition, through metadata indexing and defined recovery algorithms, the original data blocks can be effectively recovered from the compressed storage matrix to ensure data integrity and availability. The structured design of the storage matrix also facilitates parallel processing, which can easily perform parallel calculations during data processing, especially in the case of large-scale data sets, which improves the processing speed. Finally, the feature form of the storage matrix makes subsequent machine learning or data mining tasks more efficient and convenient because the features have been clearly standardized and organized.

[0035] like Figure 4 As shown, the storage matrix is ​​defined, where the matrix rows represent data blocks and the columns represent features, and the data is filled into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and the final storage matrix is ​​extracted, which specifically includes: S310, defining a storage matrix structure, determining the number of rows to be the number of merged data blocks, and each row corresponds to a data block; determining the number of columns to be the number of features, and each column corresponds to a feature; S320, creating an empty storage matrix according to a predefined dimension ; S330, traversing the merged data block features one by one ,in, Indicates samples, Indicates Features, find the corresponding feature column for each data block, and then fill the data block feature value to the corresponding position of the storage matrix to accurately map the extracted feature data to the specified position of the storage matrix to obtain the filled storage matrix: ; S340, reading all filled matrix positions in the storage matrix, and filling unfilled positions with default values ​​to form a final storage matrix.

[0036] S400, creating a metadata index, defining a recovery algorithm based on the acquired storage matrix, and decoding according to the stored metadata and splicing logic to acquire the original data block.

[0037] This step first requires extracting the location and specific values ​​of all corresponding eigenvalues ​​for each row. This process requires traversing the storage matrix row by row, identifying the features of each data block, and recording its eigenvalue and its column index in the matrix. Such operations not only help the system track the specific value of each feature, but also provide the necessary data foundation for subsequent reconstruction.

[0038] Next, according to the splicing logic extracted from the metadata index, the feature values ​​of each row after feature extraction are sorted and spliced. At this time, the splicing logic is based on the rules defined in step S200 to ensure that the reconstructed data block can faithfully reflect the structure and content of the original data. This is a critical link because inaccurate splicing may cause data distortion or loss.

[0039] After the splicing is completed, the reconstructed result is stored in the previously initialized storage space and corresponds to the original identifier or index of the data block. This correspondence not only facilitates subsequent data management, but also facilitates data retrieval and use, ensuring that the corresponding original data can be quickly located when needed.

[0040] Finally, during the data recovery process, the system compares the reconstructed data to verify the consistency of the decoded entire block with the original data. This verification step is critical and can effectively confirm the integrity and accuracy of the data and ensure that no errors or omissions were introduced during the recovery process. If inconsistencies are found, the system can make adjustments based on metadata indexing and splicing logic to optimize the recovery process.

[0041] By extracting feature values ​​row by row and reconstructing them according to the splicing logic, the system can recover the original data with minimal overhead. This approach not only improves the accuracy of data recovery, but also reduces the risk of data damage or loss. In addition, storing the correspondence with the original identifier or index makes data retrieval and management more convenient, thereby improving data availability and flexibility. At the same time, the process of verifying decoding consistency ensures the integrity of the data after recovery, laying a solid foundation for subsequent data processing and analysis.

[0042] like Figure 5 As shown, the metadata index is created, and a recovery algorithm is defined based on the acquired storage matrix, and decoding is performed according to the stored metadata and splicing logic to obtain the original data block, specifically including: S410, for each row, extracting the locations and specific values ​​of all corresponding eigenvalues; S420, sorting and splicing the feature values ​​of the row after feature extraction according to the splicing logic extracted from the metadata index, and reconstructing them into an original data block; S430, storing the spliced ​​result in a previously initialized storage space, corresponding one-to-one with the original identifier or index of the data block; S440, comparing the data during recovery to verify the consistency of the decoded entire block with the original data.

[0043] Figure 6 A structural block diagram of a data storage system based on data compression technology provided by an embodiment of the present invention, such as Figure 6 As shown, a data storage system based on data compression technology is characterized in that the system includes: The data reading and similarity analysis module 100 is used to read the data content, perform similarity analysis on the data, set a similarity threshold, and identify and obtain similar but not completely duplicated similar data blocks in the data; The data type reading and splicing module 200 is used to read the acquired data type, select a splicing rule according to the data type, splice the identified similar data blocks to obtain a spliced ​​merged block, and convert the merged block into a feature representation; The storage matrix generation module 300 is used to define a storage matrix, in which the rows of the matrix represent data blocks and the columns represent features, and fill the data into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and extract the final storage matrix; The metadata index and recovery module 400 is used to create a metadata index, define a recovery algorithm based on the acquired storage matrix, and perform decoding according to the stored metadata and splicing logic to obtain the original data block.

[0044] like Figure 7 As shown, the data reading and similarity analysis module 100 includes: The data reading unit 110 is used to read the data set and identify the data format; A duplicate data identification unit 120 is used to identify duplicate data in a data set, mark these duplicate items with indexes to obtain a number of data blocks, and count the number of repetitions and related features of each data block; A feature vector construction unit 130 is used to construct a feature vector for each data block based on the relevant features of each data block, calculate the similarity between all data blocks, and generate a similarity matrix; The similarity screening unit 140 is used to set a similarity threshold, screen the similarity matrix based on the similarity threshold, identify similar data blocks that meet the similarity threshold, and generate a similar data set list, wherein the similar data set list only includes similar data blocks that meet the similarity threshold.

[0045] like Figure 8 As shown, the data type reading and splicing module 200 includes: The data block classification unit 210 is used to classify the identified similar data blocks, define a splicing rule for each category, and perform an actual splicing operation on the identified similar data blocks according to the selected rule to obtain a merged block after splicing; A feature extraction unit 220 is used to extract key features from the merged blocks after splicing, and uniformly convert the extracted features into a vector format; The similarity evaluation unit 230 is used to evaluate the structure and logic after splicing and compare the similarity with the original data.

[0046] like Fig. 9 As shown, the storage matrix generation module 300 includes: The matrix structure definition unit 310 is used to define the storage matrix structure, determine the number of rows as the number of merged data blocks, and each row corresponds to a data block; determine the number of columns as the number of features, and each column corresponds to a feature; The empty matrix creation unit 320 is used to create an empty storage matrix according to a predefined dimension. ; The matrix filling unit 330 is used to traverse the merged data block features one by one ,in, Indicates samples, Indicates Features, find the corresponding feature column for each data block, and then fill the data block feature value to the corresponding position of the storage matrix to accurately map the extracted feature data to the specified position of the storage matrix to obtain the filled storage matrix: ; The matrix completion unit 340 is used to read all filled matrix positions in the storage matrix, and fill the unfilled positions with default values ​​to form a final storage matrix.

[0047] like Fig.10 As shown, the metadata indexing and recovery module 400 includes: The feature content extraction unit 410 is used to extract the positions and specific values ​​of all corresponding feature values ​​for each row; A splicing logic reconstruction unit 420, used to sort and splice the feature values ​​of the row after feature extraction according to the splicing logic extracted from the metadata index, and reconstruct it into an original data block; A result storage unit 430, for storing the spliced ​​result in a previously initialized storage space, in one-to-one correspondence with the original identifier or index of the data block; The consistency verification unit 440 is used to compare the data during recovery to verify the consistency of the decoded entire block with the original data.

[0048] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0049] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0050] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data storage method based on data compression technology, characterized in that: The method comprises: Read the data content, perform similarity analysis on the data, set a similarity threshold, identify and obtain similar but not completely duplicated data blocks in the data; Read the acquired data type, select a splicing rule according to the data type, splice the identified similar data blocks, obtain a spliced ​​merged block, and convert the merged block into a feature representation; Define a storage matrix, where the rows of the matrix represent data blocks and the columns represent features, and fill the data into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and extract the final storage matrix; Create a metadata index, define a recovery algorithm based on the acquired storage matrix, and decode according to the stored metadata and splicing logic to obtain the original data block.

2. The method according to claim 1, characterized in that The similarity analysis of the data, setting a similarity threshold, and identifying and obtaining similar but not completely duplicated similar data blocks in the data specifically includes: Read the data set and identify the data format; Identify duplicate data in the data set, mark these duplicates with indexes to obtain several data blocks, and count the number of repetitions and related features of each data block; Based on the relevant features of each data block, construct the feature vector of each data block, calculate the similarity of all data blocks pairwise, and generate a similarity matrix; A similarity threshold is set, and the similarity matrix is ​​screened based on the similarity threshold, similar data blocks that meet the similarity threshold are identified, and a similar data set list is generated, wherein the similar data set list only includes similar data blocks that meet the similarity threshold.

3. The method according to claim 2, characterized in that The method selects a splicing rule according to the data type, splices the identified similar data blocks to obtain a spliced ​​merged block, and converts the merged block into a feature representation, specifically including: Classify the identified similar data blocks, define splicing rules for each category, and perform actual splicing operations on the identified similar data blocks according to the selected rules to obtain the spliced ​​merged blocks; Extract key features from the merged blocks and convert the extracted features into vector format; Evaluate the structure and logic of the spliced ​​data and compare their similarity with the original data.

4. The method according to claim 3, characterized in that The storage matrix is ​​defined, wherein the rows of the matrix represent data blocks and the columns represent features, and the data is filled into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and the final storage matrix is ​​extracted, specifically including: Define the storage matrix structure, determine the number of rows as the number of merged data blocks, and each row corresponds to a data block; determine the number of columns as the number of features, and each column corresponds to a feature; Create an empty storage matrix according to predefined dimensions ; Traverse the merged data block features one by one ,in, Indicates samples, Indicates Features, find the corresponding feature column for each data block, and then fill the data block feature value to the corresponding position of the storage matrix to accurately map the extracted feature data to the specified position of the storage matrix to obtain the filled storage matrix: ; Read all filled matrix positions in the storage matrix, and fill the unfilled positions with default values ​​to form the final storage matrix.

5. The method according to claim 4, characterized in that The step of creating a metadata index, defining a recovery algorithm based on the acquired storage matrix, and decoding according to the stored metadata and splicing logic to obtain the original data block specifically includes: For each row, extract the location and specific values ​​of all corresponding eigenvalues; According to the splicing logic extracted from the metadata index, the feature values ​​of the row after feature extraction are sorted and spliced ​​to reconstruct the original data block; The concatenated result is stored in the previously initialized storage space, corresponding one-to-one to the original identifier or index of the data block; This data is compared upon recovery to verify the consistency of the decoded entire block with the original data.

6. A data storage system based on data compression technology, characterized in that: The system comprises: The data reading and similarity analysis module is used to read the data content, perform similarity analysis on the data, set the similarity threshold, and identify and obtain similar but not completely repeated similar data blocks in the data; The data type reading and splicing module is used to read the acquired data type, select the splicing rule according to the data type, splice the identified similar data blocks, obtain the spliced ​​merged block, and convert the merged block into a feature representation; The storage matrix generation module is used to define a storage matrix, where the matrix rows represent data blocks and the columns represent features, and fill the data into the corresponding positions of the storage matrix according to the feature representation after the merged block is transformed, and extract the final storage matrix; The metadata index and recovery module is used to create a metadata index, define a recovery algorithm based on the acquired storage matrix, and decode according to the stored metadata and splicing logic to obtain the original data block.

7. The system according to claim 6, characterized in that The data reading and similarity analysis module includes: A data reading unit, used for reading a data set and identifying a data format; A repeated data identification unit is used to identify repeated data in a data set, mark these repeated items with indexes to obtain several data blocks, and count the number of repetitions and related features of each data block; A feature vector construction unit is used to construct a feature vector of each data block based on relevant features of each data block, calculate similarities between all data blocks, and generate a similarity matrix; The similarity screening unit is used to set a similarity threshold, screen the similarity matrix based on the similarity threshold, identify similar data blocks that meet the similarity threshold, and generate a similar data set list, wherein the similar data set list only includes similar data blocks that meet the similarity threshold.

8. The system according to claim 7, characterized in that The data type reading and splicing module includes: A data block classification unit is used to classify the identified similar data blocks, define a splicing rule for each category, and perform an actual splicing operation on the identified similar data blocks according to the selected rule to obtain a merged block after splicing; A feature extraction unit is used to extract key features from the merged blocks after splicing, and uniformly convert the extracted features into vector format; The similarity evaluation unit is used to evaluate the structure and logic after splicing and compare its similarity with the original data.

9. The system according to claim 8, characterized in that The storage matrix generation module comprises: The matrix structure definition unit is used to define the storage matrix structure, determine the number of rows as the number of merged data blocks, and each row corresponds to a data block; determine the number of columns as the number of features, and each column corresponds to a feature; Empty matrix creation unit, used to create an empty storage matrix according to predefined dimensions ; Matrix filling unit, used to traverse the merged data block features one by one ,in, Indicates samples, Indicates Features, find the corresponding feature column for each data block, and then fill the data block feature value to the corresponding position of the storage matrix to accurately map the extracted feature data to the specified position of the storage matrix to obtain the filled storage matrix: ; The matrix completion unit is used to read all filled matrix positions in the storage matrix, and fill the unfilled positions with default values ​​to form a final storage matrix.

10. The system according to claim 9, characterized in that The metadata indexing and recovery module includes: A feature content extraction unit, used to extract the positions and specific values ​​of all corresponding feature values ​​for each row; A splicing logic reconstruction unit, used for sorting and splicing the feature values ​​of the row after feature extraction according to the splicing logic extracted from the metadata index, and reconstructing the original data block; A result storage unit, used to store the spliced ​​result in a previously initialized storage space, in one-to-one correspondence with the original identifier or index of the data block; The consistency verification unit is used to compare this data during recovery and verify the consistency of the decoded entire block with the original data.

Citation Information

Patent Citations

  • Data compression method, data decompression method and related systems

    CN107682016A

  • Data compression method and device, electronic equipment and computer readable medium

    CN112994701A

  • Method for data indexing, electronic device and computer program product

    CN113742288A

  • Data compression method, controller, device, medium and program product

    CN115145467A

  • Data compression method and device

    CN115145884A

Cited By

  • Power utilization data compression method, power utilization data transmission system, power utilization data transmission device and electronic equipment

    CN121603012A