Data storage method, device, equipment and storage medium

Abstract

The application discloses a data storage method and device, equipment and storage medium, and relates to the technical field of data storage. The method comprises the following steps: performing redundancy detection on new data, determining the redundancy position of the new data; segmenting the new data based on the redundancy position to obtain redundancy data and independent data; comparing the redundancy data with library data in a redundancy library, determining the index value of the redundancy data when the comparison is consistent; inserting the index value into the redundancy position, fusing and compressing the index value and the independent data to obtain compressed data, and storing the compressed data on a disk. Through the above method, the repeated data in the new data is replaced, the repeated part of the new data is replaced by the index value corresponding to the repeated data, the repeated data is stored once and referenced multiple times, the storage resource occupation is reduced, the utilization rate of the storage resource is improved, and more data can be stored.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a data storage method and device, equipment and a storage medium. BACKGROUND

[0002] A database will continuously generate new data from the start of operation, and in the process of data formation, due to the repetition of work content, a certain amount of repeated data will exist in the data, that is, the database will use a large amount of storage space to save multiple identical file data or text data, thereby wasting n-1 times the storage space of the size of the repeated data itself, and cannot store other data. The cost of storage space is relatively high, so this storage method will cause a great waste, and the repeated data will also affect the data retrieval efficiency, so the existing data storage method of the database has the problem of resource waste, which needs to be solved urgently.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a data storage method, device, equipment and storage medium, which aims to solve the technical problem of a large amount of repeated data occupying storage resources in the prior art data storage, causing storage resource waste.

[0005] To achieve the above purpose, the present application provides a data storage method, which comprises:

[0006] Performing redundancy detection on the new data to determine the redundancy position of the new data;

[0007] Segmenting the new data based on the redundancy position to obtain redundant data and independent data;

[0008] Comparing the redundant data with the library data in the redundancy library, and determining the index value of the redundant data when the comparison is consistent;

[0009] Inserting the index value into the redundancy position and fusing and compressing with the independent data to obtain compressed data, and storing the compressed data in the disk.

[0010] In an embodiment, the step of performing redundancy detection on the new data to determine the redundancy position of the new data comprises:

[0011] Performing block processing and cleaning processing on the new data to obtain preprocessed data;

[0012] Determining the data features of the preprocessed data, and performing hash operation on the data features to determine the new hash value of the new data;

[0013] comparing the redundancy hash values with the new hash values one by one, and determining the redundancy position of the new data when the new hash values are consistent with the redundancy hash values, the redundancy hash values being hash values of data in the redundancy library obtained through hash operation.

[0014] In an embodiment, the step of comparing the redundancy hash values with the new hash values one by one, and determining the redundancy position of the new data when the new hash values are consistent with the redundancy hash values comprises:

[0015] comparing the new hash values with redundancy hash values one by one, and determining the redundancy data corresponding to the redundancy hash values when the new hash values are consistent with the redundancy hash values;

[0016] calculating the start character and the end character of the redundancy data corresponding to the redundancy hash values, and obtaining an edit distance according to the start character and the end character;

[0017] determining a repetition scale according to the edit distance, coinciding the first position of the repetition scale with the first character of the new data, determining the coverage data, and recording the current coverage interval when the similarity between the coverage data and the redundancy data corresponding to the redundancy hash values reaches a preset similarity threshold;

[0018] marking the start position and the end position of the coverage interval as redundancy positions

[0019] In an embodiment, the step of segmenting the new data based on the redundancy positions to obtain redundancy data and independent data comprises:

[0020] determining a position identifier of the redundancy position, and determining the start position and the end position of the position identifier;

[0021] determining the start position and the end position as cutting points, generating connection marks on both sides of the cutting points, and making the connection marks of different cutting points different from each other;

[0022] taking the connection marks as data segment headers or data segment trailers respectively to obtain a plurality of data segments;

[0023] determining the data segments between the start position and the end position as redundancy data;

[0024] determining the data segments before the start position and the data segments after the end position as independent data.

[0025] In an embodiment, the step of comparing the redundancy data with the library data in the redundancy library, and determining the index value of the redundancy data when the comparison is consistent further comprises:

[0026] determining a node relationship of the library data according to a redundancy inclusion relationship between the library data, generating a redundancy index tree according to the node relationship, and generating a tree encoding according to the library data;

[0027] encoding a node on the redundancy index tree, and determining a leaf encoding of the library data in the redundancy index tree;

[0028] combining the tree encoding and the leaf encoding to obtain a library data encoding, and filling the library data encoding into a corresponding position in an index bit to obtain an index value.

[0029] In an embodiment, the step of inserting the index value into the redundancy position and performing fusion compression with the independent data to obtain compressed data comprises:

[0030] determining a first character and a last character of the index value;

[0031] sorting the independent data and the index value according to a data content order of the new data to obtain a combined sequence;

[0032] based on the combined sequence, connecting the first character of the index value with the last character of the independent data, and / or connecting the last character of the index value with the first character of the independent data to obtain fusion data;

[0033] performing fusion compression on the fusion data to obtain compressed data.

[0034] In an embodiment, the step of performing fusion compression on the fusion data to obtain compressed data comprises:

[0035] determining an initial compression segment, the initial compression segment comprising a first compression byte and a second compression byte, the first compression byte and the second compression byte each being at least one byte;

[0036] determining an index value part in the fusion data, separating the index value to obtain a tree encoding and a leaf encoding;

[0037] filling the tree encoding into a character bit in the first compression byte, and filling the leaf encoding into a character bit in the second compression byte to obtain a compression segment;

[0038] replacing the index value part in the fusion data with the compression segment to obtain compressed data;

[0039] The step of inserting the index value into the redundancy position and performing fusion compression with the independent data to obtain compressed data, and storing the compressed data on a disk further comprises:

[0040] When reaching the preset update time node, the compressed data in the target area in the disk is traversed, and independent data in the compressed data is extracted to obtain an independent data set;

[0041] The independent data set is subjected to similarity detection to determine repeated data in the independent data set;

[0042] The degree of repetition of the repeated data is determined, and when the degree of repetition is greater than a redundancy threshold, the repeated data is marked as new redundant data, and the new redundant data is added to the redundancy library.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a data storage device, which comprises:

[0044] A redundancy detection module is configured to perform redundancy detection on the new data to determine a redundancy position of the new data;

[0045] A data segmentation module is configured to segment the new data based on the redundancy position to obtain redundant data and independent data;

[0046] A data comparison module is configured to compare the redundant data with library data in a redundancy library, and determine an index value of the redundant data when the comparison is consistent;

[0047] A data compression module is configured to insert the index value into the redundancy position, and fuse and compress the index value and the independent data to obtain compressed data, and store the compressed data in a disk.

[0048] In addition, to achieve the above-mentioned purpose, the present application also provides a data storage device, which comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data storage method as described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, the computer program being executed by a processor to implement the steps of the data storage method as described above.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, the computer program being executed by a processor to implement the steps of the data storage method as described above.

[0051] The application provides a data storage method, redundancy detection is performed on new data, a redundancy position of the new data is determined, the new data is segmented based on the redundancy position, redundancy data and independent data are obtained, the redundancy data is compared with library data in a redundancy library, when the comparison is consistent, an index value of the redundancy data is determined, the index value is inserted into the redundancy position, and the index value is fused and compressed with the independent data to obtain compressed data, and the compressed data is stored in a disk. The application can replace the repeated part of the new data with the index value corresponding to the repeated data in the database, realize one-time storage of the repeated data and multiple references, and thus improve the utilization rate of storage resources, release the storage space of the repeated data, solve the technical problem that the repeated data occupies a large amount of storage space, and improve the utilization rate of storage space, so that more data can be stored under the same storage space. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0054] Figure 1 The flowchart of the data storage method of the first embodiment of the application;

[0055] Figure 2 The segmentation schematic diagram of the new data of the first embodiment of the data storage method of the application;

[0056] Figure 3 The cutting point connection schematic diagram of the first embodiment of the data storage method of the application;

[0057] Figure 4 The redundancy library construction flowchart of the first embodiment of the data storage method of the application;

[0058] Figure 5 The redundancy inclusion schematic diagram of the first embodiment of the data storage method of the application;

[0059] Figure 6 The structure schematic diagram of a redundancy index tree of the first embodiment of the data storage method of the application;

[0060] Figure 7 The module structure schematic diagram of the data storage device of the embodiment of the application;

[0061] Figure 8 The device structure diagram of the hardware running environment involved in the data storage method in the embodiments of the present application.

[0062] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0063] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0064] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] The main solution of the embodiments of the present application is: performing redundancy detection on the new data to determine the redundancy position of the new data; segmenting the new data based on the redundancy position to obtain redundant data and independent data; comparing the redundant data with the library data in the redundancy library, and determining the index value of the redundant data when the comparison is consistent; inserting the index value into the redundancy position, and fusing and compressing the independent data to obtain compressed data, and storing the compressed data on the disk.

[0066] At present, the database will continuously generate new data from the start of operation, and in the process of data formation, the repetition of work content will cause a certain amount of repeated data in the data, that is, the database will use a large amount of storage space to save multiple same file data or text data, so that n-1 times of storage space of the size of the repeated data itself is wasted, and other data cannot be stored. The cost of storage space is relatively high, so this storage method will cause great waste.

[0067] The present application provides a solution, which determines the redundancy position of the new data by performing redundancy detection on the new data, segments the new data based on the redundancy position to obtain redundant data and independent data; compares the redundant data with the library data in the redundancy library, and determines the index value of the redundant data when the comparison is consistent; inserts the index value into the redundancy position, and fuses and compresses the independent data to obtain compressed data, and stores the compressed data on the disk, realizes the replacement of the repeated data in the new data and the database, replaces the repeated part of the new data with the index value corresponding to the repeated data, realizes the storage of the repeated data once and the reference of the repeated data multiple times, thereby reducing the storage resource occupation, improving the utilization rate of the storage resource, and storing more data.

[0068] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, or an electronic device or a data storage device capable of realizing the above functions, and the embodiment does not make specific limitations. The following takes a data storage device as an example to illustrate the embodiment and the following embodiments.

[0069] The embodiment of the application provides a data storage method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the data storage method of the application is shown in the figure.

[0070] In the embodiment, the data storage method comprises steps S10-S40:

[0071] Step S10, redundancy detection is performed on the new data to determine the redundancy position of the new data.

[0072] It should be noted that the new data refers to data that needs to be added to the database due to external business environment, and the redundancy position refers to the character interval corresponding to the redundancy part determined by the redundancy detection between the new data and the existing data in the database.

[0073] It should be understood that the redundancy detection is to traverse the data content of the new data and the existing stored data in the database character by character, mark the position where the same characters exist, determine the maximum repeated segment, and form the redundancy interval with the start position and the end position corresponding to the maximum repeated segment, and the position in the new data corresponding to the redundancy interval is the redundancy position.

[0074] In a specific implementation, when data is added to the database, the data to be added to the database is regarded as new data, and the new data is compared with the existing data in the database one by one to determine whether it coincides with the existing data in the database. In the new data, if it is determined that there is a same data part as the stored data in the database, it can be determined that there is a repeated part, and the position of the repeated part in the new data is marked, and the position is determined as the redundancy position.

[0075] In a feasible embodiment, the step of performing redundancy detection on the new data to determine the redundancy position of the new data comprises:

[0076] The new data is processed by block and cleaning to obtain preprocessed data;

[0077] The data features of the preprocessed data are determined, and the data features are subjected to hash operation to determine the new hash value of the new data;

[0078] The redundant hash value is compared with the new hash value one by one, and when the new hash value is consistent with the redundant hash value, the redundant position of the new data is determined, the redundant hash value being a hash value of data in a redundant library obtained through a hash operation.

[0079] It should be noted that the data feature refers to

[0080] In a specific implementation, the new data can be processed by blocking and cleaning, at this time, it can be first divided into smaller, manageable data blocks, and according to the composition relationship of the data, it can be divided into multiple data blocks, for example, the data is "data type - storage time - data content - operator" data stored in the database, at this time, the data can be divided into at least 4 parts, including data type, storage time, data content, operator, of course, according to the data itself, it is determined, here only as an example, this embodiment does not make any limitation. After dividing the data into multiple data blocks, the data can be cleaned to remove noise, errors, and irrelevant content in the data to ensure the accuracy of subsequent processing steps. Once the data is preprocessed, the obtained data is preprocessed data, and then the preprocessed data is extracted, including but not limited to file type, size, content mode, etc., and then the hash value of the data is calculated by using the hash operation, and the obtained hash value is determined as the new hash value of the current new data.

[0081] In determining the new hash value, a new SHA-256 hash object can be created, the extracted features are serialized into a continuous byte stream, and the serialized feature data is transmitted to the SHA-256 hash object block by block. The SHA-256 algorithm processes the input data byte by byte and constantly updates the internal state. This process can be executed in parallel to speed up the hash calculation of large files. During the processing, the hash object can be set to a set of 8 32-bit initial hash values, Then, the input message is padded to make its length equal to 448 modulo 512, the padding process is to add a '1' bit first, then follow enough '0' bits, and finally add a 64-bit original message length, the padded message is divided into 512-bit blocks, if the message length is less than 512 bits, there is only one block. For each 512-bit message block, the following operations are performed:

[0082] Starting from 16 32-bit words, it is expanded to 64 32-bit words using a specific formula The expansion formula is:

[0083]

[0084] where and are two different rotation and displacement operations, defined as:

[0085]

[0086]

[0087] denotes a cyclic right shift, denotes a logical right shift, denotes a bitwise XOR.

[0088] For each , the working variables are updated, initially set to . The main part of the compression function is a loop, where each iteration updates the variables according to the current working variables, the message word and a constant . The formulas are as follows:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] where are four auxiliary functions, defined as:

[0100]

[0101]

[0102]

[0103]

[0104] wherein, represents a bitwise AND, represents a bitwise NOT.

[0105] After all the message blocks are processed, the final working variable is added to the initial hash value to obtain the final hash value, i.e.:

[0106]

[0107] This hash value is unique, and only when two data are the same data, the hash value will appear to be repeated. Therefore, to determine the redundant data provides a basis for judgment, after having the hash value of the new data, the system will compare these hash values with the hash values in the existing "redundancy library". The "redundancy library" is a database that stores the hash values corresponding to all previously processed data. If the hash value of the new data matches a hash value in the redundancy library, then it means that the new data may be duplicated, and the redundant position of the new data can be determined. When it is found that the hash value of the new data is consistent with a hash value in the redundancy library, it means that the redundancy is found. At this time, the position information of the original data can be mapped back according to the hash value, so as to clearly indicate that the redundancy exists in which specific data block or file.

[0108] In a feasible embodiment, the step of comparing the new hash value with the redundancy hash value one by one, and determining the redundant position of the new data when the new hash value is consistent with the redundancy hash value, comprises:

[0109] comparing the new hash value with the redundancy hash value one by one, and determining the redundant data corresponding to the redundancy hash value when the new hash value is consistent with the redundancy hash value;

[0110] calculating the start character and the end character of the redundant data corresponding to the redundancy hash value, and obtaining the edit distance according to the start character and the end character;

[0111] determining the repetition scale according to the edit distance, overlapping the first position of the repetition scale with the first character of the new data, determining the coverage data, and recording the current coverage interval when the similarity between the coverage data and the redundant data corresponding to the redundancy hash value reaches a preset similarity threshold;

[0112] marking the start position and the end position of the coverage interval as the redundant position.

[0113] It should be noted that the start character and end character are the first and last characters of the redundant part, and the character interval formed between the start character and end character is the redundant data. The length of the character interval is the edit distance, and a sliding window can be generated based on the edit distance. This sliding window is defined as the repetition scale, and the length of the repetition scale is the same as the length of the redundant data.

[0114] In the implementation, since hash values ​​can only be used for equality operations, when identifying redundant data, if two identical hash values ​​are found, the target data is considered redundant. Therefore, after preprocessing and feature extraction of the new data, the hash value of the new data is calculated using the SHA-256 algorithm. This new hash value is then compared one by one with each redundant hash value in the redundancy database. If a match is found, it is considered to contain potential redundant data. After identifying redundant data, it is only certain that duplicate data exists in the current new data. To process the new data, it is also necessary to determine the position of the duplicate data within the new data. Therefore, the start and end characters of the redundant data can be determined, and the distance between the start and end characters can be calculated as the edit distance. A sliding window is generated with the edit distance as a fixed length. The sliding window is determined as the repeat scale. The first position of the repeat scale is aligned with the first character of the newly added data. At this time, the data in the covered area can be matched character by character. If the match fails, the sequence is moved to the next position until a complete match is achieved. That is, when the similarity between the covered data and the redundant data covered by the repeat scale reaches 100%, the current covered interval is recorded, and the start and end positions of the covered interval are marked as redundant positions.

[0115] Step S20: The newly added data is segmented based on the redundant positions to obtain redundant data and independent data.

[0116] It should be noted that both independent data and redundant data are components of newly added data. Data that already exists in the database is identified as redundant data, while data that does not exist in the database or does not require redundancy processing is identified as independent data.

[0117] Understandably, referring to Figure 2 , Figure 2 This is a diagram illustrating the segmentation of newly added data. When redundant positions are identified, the character markers of the newly added data can be sequentially traversed based on the markers of these redundant positions to determine the corresponding segmentation markers. At this point, the data can be divided into multiple segments, and based on the segmentation markers, it can be... The data between them was identified as redundant data. The data in between are independent data, as shown in the figure. The white area represents independent data, and the gray area represents redundant data.

[0118] In an implementation, the step of splitting the new data based on the redundant positions comprises:

[0119] determining a position identifier of the redundant position, determining a start position and an end position of the position identifier;

[0120] determining the start position and the end position as cutting points, the new data being split into multiple data segments by the cutting points;

[0121] determining the data segment between the start position and the end position as redundant data;

[0122] determining the data segment before the start position and the data segment after the end position as independent data.

[0123] In a specific implementation, in the redundancy detection process, the start position and the end position of each redundant interval have been determined. These position information are stored in a list or dictionary, each redundant interval corresponds to a unique position identifier, for example, an index number, then the redundant positions are sorted by the start position, then any overlapping or adjacent intervals are merged to avoid repeated processing of the same data, creating a list containing all cutting points, including the start and end positions of each redundant interval, and the beginning and end of the new data. Then starting from the first cutting point, the substring between each two adjacent cutting points is extracted in turn to form data segments, and the data segments are classified as redundant data or independent data according to whether the cutting point belongs to the start or end position of a certain redundant interval, and finally two lists are output: one contains all redundant data segments, and the other contains all independent data segments. At the same time, ensure that each redundant data segment is associated with the corresponding position identifier.

[0124] The step of determining the start position and the end position as cutting points, and the new data being split into multiple data segments by the cutting points comprises:

[0125] determining the start position and the end position as cutting points, and generating a connection marker on both sides of the cutting point, the connection markers of different cutting points being different from each other;

[0126] using the connection markers as the head or tail of the data segment, respectively, to obtain multiple data segments.

[0127] It should be understood that the start position and the end position are determined as cutting points, and a connection marker is generated on both sides of the cutting point, the connection markers of different cutting points being different from each other, and the connection marker is represented as , for example, Figure 3As shown, Figure 3 The connection diagram of the cutting point. The connection mark can be taken as the head and tail of the data segment, and a plurality of data segments are obtained, facilitating subsequent combination of the data segments.

[0128] In step S30, the redundant data is compared with the library data in the redundancy library, and when the comparison is consistent, the index value of the redundant data is determined.

[0129] It should be noted that the index value refers to information for expressing the position of the redundant data in the redundancy library, and the position code of the redundant data for positioning, and the redundancy library is a collection of all data in the database, wherein the data in the redundancy library are different from each other except for the redundant data with too short data length, that is, only one copy of the data is saved, and the measurement standard for the data with too short length is the index value length, which is described with reference to Figure 4 , Figure 4 The flowchart for constructing the redundancy library. When the index value length is less than the length of the redundant data, the redundant data is saved in the redundancy library, and when the index value length is greater than the length of the redundant data, the redundant data is not saved in the redundancy database.

[0130] In a specific implementation, the redundant data is compared with the library data in the redundancy library, and at this time, the values compared are the hash values corresponding to the redundant data and the library data in the redundancy library respectively. Due to the characteristics of the hash value, only when the data is completely consistent, the obtained hash values are the same, and therefore when the redundant data and the data in the redundancy library are compared in the hash value, a consistent comparison result is obtained, the index value corresponding to the library data in the corresponding redundancy library can be determined.

[0131] In a feasible implementation, the step of comparing the redundant data with the library data in the redundancy library, and determining the index value of the redundant data when the comparison is consistent, further comprises:

[0132] Determining the node relationship of the library data according to the redundant inclusion relationship between the library data, generating a redundancy index tree according to the node relationship, and generating a tree code according to the library data;

[0133] Encoding the nodes on the redundancy index tree to determine the leaf code of the library data in the redundancy index tree;

[0134] Combining the tree code and the leaf code to obtain a library data code, and filling the library data code into the corresponding positions in the index bit respectively to obtain the index value.

[0135] In a specific implementation, when the redundant data is added to the redundancy library, the redundant inclusion relationship can be determined according to the redundant inclusion relationship between the library data, which is described with reference to Figure 5 , Figure 5This is a diagram illustrating redundancy. It's understandable that the longer a data type (DataA) is, the more likely a portion of it will appear in another data type (DataB). Therefore, DataA contains DataB, and the node relationship between DataA and DataB can be determined as follows: DataA is the root node, and DataB is a leaf node. A redundant index tree is generated based on the node relationships of the redundant database using a similar method, and tree encoding is generated according to the database data. The redundancy tree is determined by the type of redundant data; different redundant trees correspond to different types of redundant data. That is, a piece of redundant data can only exist in one redundant index tree. (Refer to...) Figure 6 , Figure 6 This is a schematic diagram of a redundant index tree structure. Each node includes the hash value of redundant data, and the node is encoded. Then, the tree encoding and leaf encoding are combined to obtain the library data encoding, and the library data encoding is filled into the corresponding positions in the index bits to obtain the index value. For example, the library encoding data of node F is 11.

[0136] Step S40: Insert the index value into the redundant position, and fuse and compress it with the independent data to obtain compressed data, and store the compressed data on the disk.

[0137] It should be noted that compressed data refers to data obtained by replacing redundant data with index values ​​and combining them with the corresponding data. Compared with the original data, the data size is reduced by a certain amount, which is the difference between the amount of redundant data and the amount of index value data.

[0138] Understandably, the process of inserting an index value into a redundant position and merging it with independent data to obtain compressed data can be described as follows: determining the first and last characters of the index value; sorting the independent data and the index value according to the data content order of the newly added data to obtain a combined sequence; based on the combined sequence, concatenating the first character of the index value with the last character of the independent data, and / or concatenating the last character of the index value with the first character of the independent data to obtain fused data; and merging and compressing the fused data to obtain compressed data. Specifically, it is possible to determine the first and last characters of the index value, sort the independent data and the index value according to the data content order in the newly added data to obtain a combined sequence, then concatenate the last character of the index value with the first character of the independent data, or concatenate the last character of the index value with the first character of the independent data to obtain fused data, and then compress the fused data to obtain fused data.

[0139] In one feasible implementation, the step of fusing and compressing the fused data to obtain compressed data includes:

[0140] determining an initial compression segment, the initial compression segment comprising a first compression byte and a second compression byte, the first compression byte and the second compression byte each being at least one byte;

[0141] determining an index value part in the fusion data, separating the index value into a tree code and a leaf code;

[0142] filling the tree code into character bits in the first compression byte and filling the leaf code into character bits in the second compression byte, respectively, to obtain a compression segment;

[0143] replacing the index value part in the fusion data with the compression segment to obtain compressed data.

[0144] It should be noted that the first compression byte and the second compression byte are two components of the index value, and the byte length of the first compression byte and the second compression byte can be extended according to specific circumstances, and is 1 byte in the initial state. The compression segment is a data segment obtained after the index value is filled into the compression byte.

[0145] In a specific implementation, the initial compression segment is composed of two compression bytes, which are the first compression byte and the second compression byte. Each byte contains at least one byte of data. According to specific requirements and the characteristics of redundant data, a suitable byte length can be selected, and 8 bits are used as the length of the compression byte in this embodiment. In the fusion data, all inserted index values are found. These index values are identifiers used to replace redundant data before, and each index value is separated into two parts: a tree code and a leaf code. The tree code represents the structural information of the index value, and the leaf code represents the specific content of the index value. Then the tree code is filled into the character bits in the first compression byte, and the leaf code is filled into the character bits in the second compression byte. If the code is too long, multiple bytes are used for storage. Then the first compression byte and the second compression byte filled with the tree code and the leaf code are combined into a compression segment, and the generated compression segment is used to replace the corresponding index value part in the fusion data. In this way, the amount of data actually stored can be reduced while the data structure is preserved. All the fusion data segments after replacement are combined to form the final compressed data, which is stored in the disk or other persistent storage medium. At the same time, necessary metadata (such as an index mapping table, a compression algorithm type, etc.) is saved for subsequent decompression and recovery.

[0146] In a feasible implementation, the step of inserting the index value into the redundant position, fusing and compressing with the independent data to obtain compressed data, and storing the compressed data in the disk further comprises:

[0147] When a preset update time node is reached, the compressed data in the target area in the disk is traversed, and independent data in the compressed data is extracted to obtain an independent data set;

[0148] The independent data set is subjected to similarity detection to determine repeated data in the independent data set;

[0149] The degree of repetition of the repeated data is determined, and when the degree of repetition is greater than a redundancy threshold, the repeated data is marked as new redundant data, and the new redundant data is added to the redundancy library.

[0150] In a specific implementation, the database can periodically perform redundancy detection on data in the database. When a preset update time node is reached, for example, 03:00 on the first day of each month, the compressed data in the target area of the disk can be traversed to extract independent data therein to obtain an independent data set. Then, the data in the independent data set is subjected to similarity detection to determine repeated data in the independent data set. According to the repeated data, the degree of repetition is determined. When the degree of repetition is greater than a redundancy threshold, the repeated data can be marked as new redundant data, and the new redundant data is added to the redundancy library. Further data compression is performed on the compressed data to reduce storage space occupation.

[0151] The embodiment provides a data storage method. Redundancy detection is performed on new data to determine a redundancy position of the new data. The new data is segmented based on the redundancy position to obtain redundant data and independent data. The redundant data is compared with library data in a redundancy library. When the comparison is consistent, an index value of the redundant data is determined. The index value is inserted into the redundancy position and fused with the independent data for compression to obtain compressed data. The compressed data is stored in a disk. The data in the new data that is repeated in the database is replaced by an index value corresponding to the repeated data. The repeated data is stored once and referenced multiple times, thereby reducing storage resource occupation, improving storage resource utilization, and storing more data.

[0152] It should be noted that the above examples are only used for understanding the present application and do not constitute a limitation on the data storage method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0153] The present application also provides a data storage device. Please refer to Figure 7 The data storage device comprises:

[0154] The redundancy detection module 10 is configured to perform redundancy detection on new data to determine a redundancy position of the new data.

[0155] The data segmentation module 20 is configured to segment the new data based on the redundancy position to obtain redundancy data and independent data.

[0156] The data comparison module 30 is configured to compare the redundancy data with library data in a redundancy library, and determine an index value of the redundancy data when the comparison is consistent.

[0157] The data compression module 40 is configured to insert the index value into the redundancy position, and fuse and compress the index value and the independent data to obtain compressed data, and store the compressed data in a disk.

[0158] In an embodiment, the redundancy detection module 10 is further configured to perform block processing and cleaning processing on the new data to obtain preprocessed data, determine a data feature of the preprocessed data, perform a hash operation on the data feature to determine a new hash value of the new data, compare the new hash value with redundancy hash values one by one, and determine a redundancy position of the new data when the new hash value is consistent with the redundancy hash value. The redundancy hash value is a hash value of data in a redundancy library obtained by a hash operation.

[0159] In an embodiment, the redundancy detection module 10 is further configured to compare the new hash value with redundancy hash values one by one, and determine redundancy data corresponding to the redundancy hash value when the new hash value is consistent with the redundancy hash value. The redundancy detection module 10 is further configured to calculate a start character and an end character of the redundancy data corresponding to the redundancy hash value, obtain an edit distance according to the start character and the end character, determine a repetition scale according to the edit distance, overlap a first position of the repetition scale with a first character of the new data, determine overlapping data, record a current overlapping interval when a similarity of the overlapping data and the redundancy data corresponding to the redundancy hash value reaches a preset similarity threshold, and mark a start position and an end position of the overlapping interval as redundancy positions.

[0160] In an embodiment, the data segmentation module 20 is further configured to determine a position identifier of the redundancy position, determine a start position and an end position of the position identifier, determine a cutting point with the start position and the end position, generate a connection mark on both sides of the cutting point, and make the connection marks on both sides of different cutting points different from each other. The data segmentation module 20 is further configured to take the connection marks as a data segment head or a data segment tail to obtain a plurality of data segments, determine the data segments between the start position and the end position as redundancy data, and determine the data segments before the start position and the data segments after the end position as independent data.

[0161] In an implementation, the data comparison module 30 is further configured to determine a node relationship of the library data according to a redundant inclusion relationship between the library data, generate a redundant index tree according to the node relationship, and generate a tree code according to the library data; encode nodes on the redundant index tree, determine a leaf code of the library data in the redundant index tree; combine the tree code and the leaf code to obtain a library data code, and fill the library data code into corresponding positions in index bits to obtain an index value.

[0162] In an implementation, the data compression module 40 is further configured to determine a first character and a last character of the index value; sort the independent data and the index value according to a data content sequence of the new data to obtain a combined sequence; connect the first character of the index value with the last character of the independent data and / or connect the last character of the index value with the first character of the independent data to obtain fusion data based on the combined sequence; and perform fusion compression on the fusion data to obtain compressed data.

[0163] In an implementation, the data compression module 40 is further configured to determine an initial compression segment, the initial compression segment including a first compression byte and a second compression byte, the first compression byte and the second compression byte each being at least one byte; determine an index value part in the fusion data, separate the index value to obtain a tree code and a leaf code; fill the tree code into character bits in the first compression byte and fill the leaf code into character bits in the second compression byte to obtain a compression segment; replace the index value part in the fusion data with the compression segment to obtain compressed data; and after storing the compressed data in the disk, further include: when a preset update time node is reached, traverse the compressed data in a target area in the disk, and extract independent data in the compressed data to obtain an independent data set; perform similarity detection on the independent data set to determine repeated data in the independent data set; determine a repetition degree of the repeated data, and when the repetition degree is greater than a redundancy threshold, mark the repeated data as new redundant data, and add the new redundant data to the redundancy library.

[0164] The data storage device provided in the application adopts the data storage method in the above embodiments, and can solve the technical problem of storage resource waste caused by a large amount of repeated data occupying storage resources in data storage. Compared with the prior art, the data storage device provided in the application has the same beneficial effects as the data storage method provided in the above embodiments, and other technical features in the data storage device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0165] The application provides a data storage device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data storage method in the above embodiment one.

[0166] Reference will now be made to the following description Figure 8 which shows a structural schematic diagram of a data storage device suitable for implementing the embodiments of the application. The data storage device in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 8 The data storage device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the application.

[0167] As shown in Figure 8 , the data storage device can include a processing apparatus 1001 (for example, a central processor, a graphics processor, and the like) which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or loaded from a storage apparatus 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for operation of the data storage device are also stored. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication apparatus 1009. The communication apparatus 1009 can allow the data storage device to communicate with other devices wirelessly or by wire to exchange data. Although the data storage device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or provided. More or less systems can be alternatively implemented or provided.

[0168] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0169] The data storage device provided by the present application adopts the data storage method in the above-mentioned embodiments, and can solve the technical problem of data storage. Compared with the prior art, the data storage device provided by the present application has the same beneficial effects as the data storage method provided by the above-mentioned embodiments, and other technical features in the data storage device are the same as the features disclosed in the above-mentioned embodiments, which will not be repeated here.

[0170] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0171] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0172] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the data storage method in the above-mentioned embodiments.

[0173] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0174] The computer readable storage medium described above may be contained in a data storage device, or may exist separately without being assembled into the data storage device.

[0175] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the data storage device, the data storage device is caused to: perform redundancy detection on new data, determine a redundancy position of the new data; segment the new data based on the redundancy position to obtain redundancy data and independent data; compare the redundancy data with library data in a redundancy library, when the comparison is consistent, determine an index value of the redundancy data; insert the index value into the redundancy position, and fuse and compress the index value with the independent data to obtain compressed data, and store the compressed data in a disk.

[0176] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0177] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0178] The modules involved in the embodiments of the present application can be implemented in the manner of software or hardware. Among them, the name of the module does not constitute the limitation of the unit itself in some cases.

[0179] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the above data storage method, and can solve the technical problem of data storage. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the data storage method provided by the above embodiments, which will not be repeated here.

[0180] ​The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the data storage method as described above.

[0181] The computer program product provided by the application can solve the technical problem of data storage. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the data storage method provided by the above-mentioned embodiments, and are not described here.

[0182] The above is only some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation made by using the content of the specification and drawings of the application, or direct / indirect application in other related technical fields is included in the patent protection scope of the application.

Claims

1. A data storage method, characterized by, The data storage method comprises: performing redundancy detection on the new data to determine a redundancy position of the new data; segmenting the new data based on the redundancy position to obtain redundant data and independent data; comparing the redundant data with library data in a redundancy library, and determining an index value of the redundant data when the comparison is consistent; inserting the index value into the redundancy position and performing fusion compression on the index value and the independent data to obtain compressed data, and storing the compressed data in a disk; the step of inserting the index value into the redundancy position and performing fusion compression on the index value and the independent data to obtain compressed data comprises: determining a first character and a tail character of the index value; sorting the independent data and the index value according to the data content order of the new data to obtain a combined sequence; based on the combined sequence, connecting the first character of the index value with the tail character of the independent data, and / or connecting the tail character of the index value with the first character of the independent data to obtain fusion data; performing fusion compression on the fusion data to obtain compressed data; the step of performing fusion compression on the fusion data to obtain compressed data comprises: determining an initial compression segment, the initial compression segment comprising a first compression byte and a second compression byte, the first compression byte and the second compression byte being at least one byte respectively; determining an index value part in the fusion data, separating the index value to obtain tree encoding and leaf encoding; filling the tree encoding into character bits in the first compression byte and filling the leaf encoding into character bits in the second compression byte to obtain a compression segment; replacing the index value part in the fusion data with the compression segment to obtain compressed data; the step of inserting the index value into the redundancy position and performing fusion compression on the index value and the independent data to obtain compressed data, and storing the compressed data in a disk further comprises: when a preset update time node is reached, traversing the compressed data in a target area in the disk and extracting independent data in the compressed data to obtain an independent data set; performing similarity detection on the independent data set to determine repeated data in the independent data set; determining a repetition degree of the repeated data, and when the repetition degree is greater than a redundancy threshold, marking the repeated data as new redundant data and adding the new redundant data to the redundancy library.

2. The method of claim 1, wherein, the step of performing redundancy detection on the new data to determine a redundancy position of the new data comprises: performing block processing and cleaning processing on the new data to obtain preprocessed data; determining data features of the preprocessed data and performing hash operation on the data features to determine a new hash value of the new data; comparing redundancy hash values with the new hash value one by one, and when the new hash value is consistent with the redundancy hash value, determining a redundancy position of the new data, the redundancy hash value being a hash value of data in a redundancy library obtained through hash operation.

3. The method of claim 2, wherein, The step of comparing the redundant hash value with the new hash value one by one, and determining the redundant position of the new data when the new hash value is consistent with the redundant hash value, comprises: comparing the redundant hash value with the new hash value one by one, and determining the redundant data corresponding to the redundant hash value when the new hash value is consistent with the redundant hash value; calculating the start character and the end character of the redundant data corresponding to the redundant hash value, and obtaining the edit distance according to the start character and the end character; determining the repetition scale according to the edit distance, overlapping the first position of the repetition scale with the first character of the new data, determining the coverage data, and recording the current coverage interval when the similarity between the coverage data and the redundant data corresponding to the redundant hash value reaches a preset similarity threshold; marking the start position and the end position of the coverage interval as the redundant position.

4. The method of claim 1, wherein, The step of segmenting the new data based on the redundant position to obtain redundant data and independent data comprises: determining the position identifier of the redundant position, determining the start position and the end position of the position identifier; determining the start position and the end position as cutting points, generating connection marks on both sides of the cutting points, and the connection marks of different cutting points are different from each other; using the connection marks as data segment headers or data segment trailers respectively to obtain a plurality of data segments, and the new data is segmented into a plurality of data segments by the cutting points; determining the data segment between the start position and the end position as the redundant data; determining the data segment before the start position and the data segment after the end position as the independent data.

5. The method of claim 1, wherein, The step of comparing the redundant data with the library data in the redundant library, and determining the index value of the redundant data when the comparison is consistent, further comprises: determining the node relationship of the library data according to the redundant inclusion relationship between the library data, generating a redundant index tree according to the node relationship, and generating a tree encoding according to the library data; encoding the nodes on the redundant index tree to determine the leaf encoding of the library data in the redundant index tree; combining the tree encoding and the leaf encoding to obtain a library data encoding, and filling the library data encoding into the corresponding position in the index bit to obtain the index value.

6. A data storage device, characterized by The data storage device comprises: a redundancy detection module for detecting the redundancy of new data and determining the redundant position of the new data; a data segmentation module for segmenting the new data based on the redundant position to obtain redundant data and independent data; a data comparison module for comparing the redundant data with the library data in the redundant library, and determining the index value of the redundant data when the comparison is consistent; a data compression module for inserting the index value into the redundant position, and fusing and compressing the independent data to obtain compressed data, and storing the compressed data on the disk; The step of inserting the index value into the redundant position and fusing and compressing the independent data to obtain compressed data comprises: determining the first character and the last character of the index value; sequencing the independent data and the index value according to data content order of the new data to obtain a combined sequence; based on the combined sequence, connecting a first character of the index value with a last character of the independent data, and / or connecting a last character of the index value with a first character of the independent data to obtain fusion data; fusion compressing the fusion data to obtain compressed data; the step of fusion compressing the fusion data to obtain compressed data comprises: determining an initial compression segment, the initial compression segment comprising a first compression byte and a second compression byte, the first compression byte and the second compression byte being at least one byte respectively; determining an index value part in the fusion data, separating the index value to obtain tree encoding and leaf encoding; filling the tree encoding into character bits in the first compression byte and filling the leaf encoding into character bits in the second compression byte respectively to obtain a compression segment; replacing the index value part in the fusion data with the compression segment to obtain compressed data; the step of inserting the index value into the redundant position, fusion compressing the independent data to obtain compressed data, and storing the compressed data in the disk further comprises: when a preset update time node is reached, traversing the compressed data in a target area in the disk and extracting independent data in the compressed data to obtain an independent data set; detecting similarity of the independent data set to determine repeated data in the independent data set; determining a degree of repetition of the repeated data, when the degree of repetition is greater than a redundancy threshold, marking the repeated data as new redundant data, and adding the new redundant data to the redundancy library.

7. A data storage device, characterized by The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the data storage method according to any one of claims 1 to 5.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, which is executed by a processor to implement the steps of the data storage method according to any one of claims 1 to 5.

Images