Data storage method and device, equipment and storage medium

Abstract

The invention discloses a data storage method and device, equipment and a storage medium, and relates to the technical field of data storage, the method comprises the following steps: carrying out redundancy detection on newly added data, and determining a redundancy position of the newly added data; segmenting the newly added data based on the redundant position to obtain redundant data and independent data; comparing the redundant data with library data in a redundant library, and determining an index value of the redundant data when the redundant data is consistent with the library data in the redundant library; and inserting the index value into the redundant position, performing fusion compression on the index value and the independent data to obtain compressed data, and storing the compressed data in a disk. Through the mode, the repeated data in the newly-added data and the database are replaced, the index value corresponding to the repeated data is used for replacing the repeated part of the newly-added data, and the repeated data are stored once and referenced for multiple times, so that the occupation of storage resources is reduced, the utilization rate of the storage resources is improved, and more data are stored.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to data storage methods, devices, equipment and storage media. Background Art

[0002] The database will continuously generate new data from the beginning of operation. In the process of data formation, due to the repetitiveness of work content, there will be a certain amount of duplicate data in the data. That is to say, the database will use a large amount of storage space to save multiple identical file data or text data, so that the storage space of n-1 times the size of the duplicate data itself is wasted and other data cannot be stored. The cost of storage space is high, so this storage method will cause a great degree of waste, and duplicate data will also affect data retrieval efficiency. Therefore, the existing database data storage method has a problem of resource waste that needs to be solved urgently.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a data storage method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that a large amount of duplicate data occupies storage resources and causes waste of storage resources.

[0005] To achieve the above purpose, the present application provides a data storage method, the method comprising: Performing redundancy detection on the newly added data to determine the redundant position of the newly added data; Segmenting the newly added data based on the redundant position to obtain redundant data and independent data; Comparing the redundant data with the library data in the redundant library, and when the comparison is consistent, determining the index value of the redundant data; The index value is inserted into the redundant position, and is fused and compressed with the independent data to obtain compressed data, and the compressed data is stored in a disk.

[0006] In one embodiment, the step of performing redundancy detection on the newly added data and determining the redundant position of the newly added data includes: The newly added data is divided into blocks and cleaned to obtain pre-processed data; Determining data features of the preprocessed data, and performing a hash operation on the data features to determine a newly added hash value of the newly added data; The redundant hash value is compared with the newly added hash value one by one, and when the newly added hash value is consistent with the redundant hash value, the redundant position of the newly added data is determined, and the redundant hash value is a hash value obtained by hashing the data in the redundant library.

[0007] In one embodiment, the step of comparing the redundant hash value with the newly added hash value one by one, and when the newly added hash value is consistent with the redundant hash value, determining the redundant position of the newly added data includes: Compare the newly added hash value with the redundant hash value one by one, and when the newly added hash value is consistent with the redundant hash value, determine the redundant data corresponding to the redundant hash value; Calculate the starting character and the ending character of the redundant data corresponding to the redundant hash value, and obtain the edit distance according to the starting character and the ending character; Determine a repetition scale according to the edit distance, overlap the first position of the repetition scale with the first character of the newly added data, determine the coverage data, and record the current coverage interval when the similarity between the coverage data and the redundant data corresponding to the redundant hash value reaches a preset similarity threshold; Mark the start and end positions of the coverage interval as redundant positions In one embodiment, the step of segmenting the newly added data based on the redundant position to obtain redundant data and independent data includes: Determine a position identifier of the redundant position, and determine a starting position and an ending position of the position identifier; The starting position and the ending position are determined as cutting points, and characters before and after the cutting points are used to generate connection marks, wherein the connection marks of different cutting points are different from each other; Using the connection mark as a data segment header or a data segment tail, respectively, to obtain multiple data segments; Determine the data segment between the starting position and the ending position as redundant data; The data segment before the start position and the data segment after the end position are determined as independent data.

[0008] In one embodiment, the step of comparing the redundant data with the library data in the redundant library, and when the comparison is consistent, before determining the index value of the redundant data, further includes: Determine the node relationship of the library data according to the redundant inclusion relationship between the library data, generate a redundant index tree according to the node relationship, and generate a tree code according to the library data; Encoding the nodes on the redundant index tree to determine the leaf codes of the library data in the redundant index tree; The tree code and the leaf code are combined to obtain a library data code, and the library data code is filled into corresponding positions in the index bit positions to obtain an index value.

[0009] In one embodiment, the step of inserting the index value into the redundant position and fusing and compressing it with the independent data to obtain compressed data includes: Determine the first and last characters of the index value; Sorting the independent data and the index values ​​according to the data content order of the newly added data to obtain a combined sequence; Based on the combination sequence, the first character of the index value is connected with the last character of the independent data, and / or the last character of the index value is connected with the first character of the independent data to obtain fused data; The fused data is fused and compressed to obtain compressed data.

[0010] In one embodiment, the step of fusing and compressing the fused data to obtain compressed data includes: Determine an initial compressed segment, the initial compressed segment comprising a first compressed byte and a second compressed byte, the first compressed byte and the second compressed byte are at least one byte respectively; Determine an index value portion in the fused data, and separate the index value into a tree code and a leaf code; Fill the tree codes into the character bits in the first compressed byte respectively, and fill the leaf codes into the character bits in the second compressed byte respectively, to obtain compressed fragments; Replacing the index value part in the fused data with the compressed fragment to obtain compressed data; After the step of inserting the index value into the redundant position and fusing and compressing it with the independent data to obtain compressed data, and storing the compressed data on a disk, the method further includes: When a preset update time node is reached, the compressed data in the target area of ​​the disk is traversed, and independent data in the compressed data is extracted to obtain an independent data set; Performing similarity detection on the independent data set to determine duplicate data in the independent data set; The duplication degree of the duplicate data is determined, and when the duplication degree is greater than a redundancy threshold, the duplicate data is marked as new redundant data, and the new redundant data is added to the redundant library.

[0011] In addition, to achieve the above-mentioned purpose, the present application also proposes a data storage device, the data storage device comprising: A redundancy detection module, used to perform redundancy detection on the newly added data and determine the redundant position of the newly added data; A data segmentation module, used for segmenting the newly added data based on the redundant position to obtain redundant data and independent data; A data comparison module, used to compare the redundant data with the library data in the redundant library, and when the comparison is consistent, determine the index value of the redundant data; The data compression module is used to insert the index value into the redundant position, merge and compress it with the independent data to obtain compressed data, and store the compressed data on a disk.

[0012] In addition, to achieve the above objectives, the present application also proposes a data storage device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data storage method as described above.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data storage method as described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data storage method as described above are implemented.

[0015] The present application provides a data storage method, which performs redundancy detection on newly added data, determines the redundant position of the newly added data, and divides the newly added data based on the redundant position to obtain redundant data and independent data; compares the redundant data with the library data in the redundant library, and when the comparison is consistent, determines the index value of the redundant data; inserts the index value into the redundant position, and merges and compresses it with the independent data to obtain compressed data, and stores the compressed data on a disk. The present application replaces the duplicated data in the newly added data with the index value corresponding to the duplicated data, thereby realizing one-time storage and multiple references of the duplicated data, thereby improving the utilization rate of storage resources, releasing the storage space of the duplicated data, solving the technical problem that duplicated data occupies a large amount of storage space, improving the utilization rate of storage space, and being able to store more data in the same storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 This is a flow chart of Embodiment 1 of the data storage method of the present application; Figure 2 This is a schematic diagram of newly added data segmentation in an embodiment of the data storage method of the present application; Figure 3 This is a schematic diagram of the cutting point connection of an embodiment of the data storage method of the present application; Figure 4 A flowchart of redundant library construction in an embodiment of a data storage method of the present application; Figure 5 A redundancy diagram of an embodiment of a data storage method of the present application; Figure 6 A schematic diagram of the structure of a redundant index tree in an embodiment of the data storage method of the present application; Figure 7 This is a schematic diagram of the module structure of the data storage device according to an embodiment of the present application; Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the data storage method in the embodiment of the present application.

[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of the embodiment of the present application is: perform redundancy detection on the newly added data to determine the redundant position of the newly added data; segment the newly added data based on the redundant position to obtain redundant data and independent data; compare the redundant data with the library data in the redundant library, and when the comparison is consistent, determine the index value of the redundant data; insert the index value into the redundant position, and merge and compress it with the independent data to obtain compressed data, and store the compressed data on the disk.

[0023] At present, databases will continuously generate new data from the beginning of operation. In the process of data formation, due to the repetitiveness of work content, there will be a certain amount of duplicate data in the data. That is to say, the database will use a large amount of storage space to save multiple identical file data or text data, so that the storage space of n-1 times the size of the duplicate data itself is wasted and other data cannot be stored. The cost of storage space is high, so this storage method will cause a great degree of waste.

[0024] The present application provides a solution, which performs redundancy detection on newly added data to determine the redundant position of the newly added data, and divides the newly added data based on the redundant position to obtain redundant data and independent data; compares the redundant data with the library data in the redundant library, and when the comparison is consistent, determines the index value of the redundant data; inserts the index value into the redundant position, and merges and compresses it with the independent data to obtain compressed data, and stores the compressed data on a disk, so as to replace the data in the newly added data that is repeated in the database, and replaces the repeated part of the newly added data with the index value corresponding to the repeated data, so as to store the repeated data once and reference it multiple times, thereby reducing the storage resource occupancy, improving the utilization rate of the storage resources, and storing more data.

[0025] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a data storage device, etc. that can realize the above functions, and this embodiment does not specifically limit this. The following takes a data storage device as an example to illustrate this embodiment and the following embodiments.

[0026] The present application embodiment provides a data storage method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data storage method of the present application.

[0027] In this embodiment, the data storage method includes steps S10 to S40: Step S10: performing redundancy detection on the newly added data to determine the redundant position of the newly added data.

[0028] It should be noted that the newly added data refers to the data that needs to be added to the database due to the external business environment, and the redundant position refers to the character interval corresponding to the redundant part determined by redundancy detection between the newly added data and the data already existing in the database.

[0029] It should be understood that redundancy detection is to traverse the data content of the newly added data and the existing stored data in the database character by character, mark the positions where the same characters exist, determine the largest repeated segment, and form a redundant interval with the start position and end position corresponding to the largest repeated segment. The position in the newly added data corresponding to the redundant interval is the redundant position.

[0030] In a specific implementation, when data is added to a database, the data to be added to the database is considered as newly added data, and the newly added data is compared with the data in the database one by one to determine whether it overlaps with the existing data in the database. When it is determined that there is a data portion in the newly added data that is the same as the data stored in the database, it can be determined that there is a duplicate portion, and the position of the duplicate portion in the newly added data is marked, and the position is determined as a redundant position.

[0031] In a feasible embodiment, the step of performing redundancy detection on the newly added data and determining the redundant position of the newly added data includes: The newly added data is divided into blocks and cleaned to obtain pre-processed data; Determining data features of the preprocessed data, and performing a hash operation on the data features to determine a newly added hash value of the newly added data; The redundant hash value is compared with the newly added hash value one by one, and when the newly added hash value is consistent with the redundant hash value, the redundant position of the newly added data is determined, and the redundant hash value is a hash value obtained by hashing the data in the redundant library.

[0032] It should be noted that data features refer to In a specific implementation, the newly added data can be processed in blocks and cleaned. At this time, it can be first divided into smaller, manageable data blocks, and can be divided into multiple data blocks according to the composition relationship of the data. For example, the data of "data type-storage time-data content-operator" is stored in the database. At this time, the data can be divided into at least 4 parts, including at least data type, storage time, data content, and operator. Of course, it is determined specifically according to the data itself. This is only an example, and this embodiment does not limit this. After the data is divided into multiple data blocks, the data can be cleaned to remove noise, errors, and irrelevant content in the data to ensure the accuracy of subsequent processing steps. Once the data is preprocessed, the data obtained is preprocessed data, and feature extraction is performed on the preprocessed data, including but not limited to file type, size, content mode, etc., and then the hash value of these data is calculated using hash operation, and the obtained hash value is determined as the newly added hash value of the current newly added data.

[0033] When the new hash value is determined, a new SHA-256 hash object can be created, the extracted features are serialized and converted into a continuous byte stream, and the serialized feature data is passed to the SHA-256 hash object block by block. The SHA-256 algorithm processes the input data byte by byte and continuously updates the internal state. This process can be performed in parallel to speed up the hash calculation of large files. During the process, the hash object can be set to a set of 8 32-bit initial hash values, , and then pad the input message to make its length equal to 448 modulo 512. The padding process is to first add a '1' bit, followed by enough '0' bits, and finally add a 64-bit original message length. The padded message is divided into 512-bit blocks. If the message length is less than 512 bits, there is only one block. For each 512-bit message block, perform the following operations: Starting with 16 32-bit words, it expands to 64 32-bit words using a specific formula , the expanded formula is:

[0034] in, and They are two different rotation and translation operations, defined as:

[0035]

[0036] Indicates circular right shift, Indicates logical right shift, Represents bitwise exclusive OR.

[0037] For each , update the working variables , the initial setting is The main part of the compression function is a loop, in which each iteration is based on the current working variables, message characters, and a constant To update these variables. The formula is as follows:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] in, There are four auxiliary functions, namely:

[0048]

[0049]

[0050]

[0051] in, represents bitwise AND, Indicates bitwise inversion.

[0052] After all message blocks are processed, the final working variable Add to the initial hash value The final hash value is:

[0053] This hash value is unique, and only when two data are the same data will the hash value be repeated. Therefore, in order to provide a basis for determining redundant data, after having the hash value of the newly added data, the system will compare these hash values ​​with the hash values ​​in the existing "redundant library". The "redundant library" is a database that stores the hash values ​​corresponding to all previously processed data. If the hash value of the newly added data matches a hash value in the redundant library, it means that the newly added data may be duplicated, and the redundant position of the newly added data can be determined. When the hash value of the newly added data is found to be consistent with a hash value in the redundant library, it means that redundancy has been found. At this time, the location information of the original data can be mapped back based on the hash value, so as to clearly indicate in which specific data block or file the redundancy exists.

[0054] In a feasible embodiment, the step of comparing the newly added hash value with the redundant hash value one by one, and when the newly added hash value is consistent with the redundant hash value, determining the redundant position of the newly added data includes: Compare the newly added hash value with the redundant hash value one by one, and when the newly added hash value is consistent with the redundant hash value, determine the redundant data corresponding to the redundant hash value; Calculate the starting character and the ending character of the redundant data corresponding to the redundant hash value, and obtain the edit distance according to the starting character and the ending character; Determine a repetition scale according to the edit distance, overlap the first position of the repetition scale with the first character of the newly added data, determine the coverage data, and record the current coverage interval when the similarity between the coverage data and the redundant data corresponding to the redundant hash value reaches a preset similarity threshold; The start position and the end position of the coverage interval are marked as redundant positions.

[0055] It should be noted that the start character and the end character are the first character and the last character of the redundant part, and the character interval formed between the start character and the end character is redundant data. The length of the character interval is the edit distance, and a sliding window can be generated according to the edit distance. The sliding window is defined as the repetition scale, and the length of the repetition scale is consistent with the length of the redundant data.

[0056] In the specific implementation, since hash values ​​can only perform equivalent operations, when judging redundant data, when two identical hash values ​​are determined, the target data is determined to be redundant data. Therefore, after preprocessing and feature extraction of the newly added data, the hash value of the newly added data can be calculated using the SHA-256 algorithm, and the newly added hash value is compared with each redundant hash value in the redundant library one by one. If a match is found, it is considered that there is potential redundant data. After the redundant data is determined, it can only be determined that there is duplicate data in the current newly added data. In order to process the newly added data, it is also necessary to determine the position of the duplicate data in the newly added data. Therefore, the start and end characters of the redundant data can be determined, and the distance between the start character and the end character can be determined, and the edit distance is obtained by the distance. Generate a sliding window with edit distance as fixed length, determine the sliding window as the repetition scale, and overlap the first position of the repetition scale with the first character of the newly added data. At this time, the data in the coverage area can be matched character by character. When the match fails, the sequence is moved back until it is completely matched. That is, when the similarity between the coverage data covered by the repetition scale and the redundant data reaches 100%, the current coverage interval is recorded, and the start and end positions of the coverage interval are marked as redundant positions.

[0057] Step S20, dividing the newly added data based on the redundant position to obtain redundant data and independent data.

[0058] It should be noted that both independent data and redundant data are components of newly added data. Data that currently exists in the database is determined as redundant data, while data that does not exist in the database or does not need to be processed redundantly is determined as independent data.

[0059] It is understandable that, referring to Figure 2 , Figure 2 When the redundant position is obtained, the character mark bits of the newly added data can be sequentially traversed based on the mark of the redundant position to determine the corresponding segmentation mark. At this time, the data can be cut into multiple segments, and according to the segmentation mark, The data between is determined as redundant data. The data in between are independent data. For example, the white area in the figure is independent data, and the gray area is redundant data.

[0060] In a feasible implementation manner, the step of segmenting the newly added data based on the redundant position to obtain redundant data and independent data includes: Determine a position identifier of the redundant position, and determine a starting position and an ending position of the position identifier; The starting position and the ending position are determined as cutting points, and the newly added data is divided into multiple data segments by the cutting points; Determine the data segment between the starting position and the ending position as redundant data; The data segment before the start position and the data segment after the end position are determined as independent data.

[0061] In a specific implementation, during the redundancy detection process, the starting position and the ending position of each redundant interval have been determined. These position information are stored in a list or dictionary, and each redundant interval corresponds to a unique position identifier, such as an index number. Then the redundant positions are sorted by the starting position, and then any overlapping or adjacent intervals are merged to avoid repeated processing of the same data. A list containing all cutting points is created, including the starting and ending positions of each redundant interval, as well as the beginning and end of the newly added data. Then, starting from the first cutting point, the substrings between each two adjacent cutting points are extracted in sequence to form data fragments. According to whether the cutting point belongs to the starting or ending position of a redundant interval, the data fragments are classified as redundant data or independent data. Finally, two lists are output: one contains all redundant data fragments and the other contains all independent data fragments. At the same time, it is ensured that each redundant data fragment is associated with the corresponding position identifier.

[0062] The step of determining the starting position and the ending position as cutting points, and dividing the newly added data into a plurality of data segments by the cutting points comprises: The starting position and the ending position are determined as cutting points, and characters before and after the cutting points are used to generate connection marks, wherein the connection marks of different cutting points are different from each other; The connection mark is used as a data segment header or a data segment tail to obtain multiple data segments.

[0063] It should be understood that the starting position and the ending position are determined as the cutting points, and the characters before and after the cutting points generate connection marks. The connection marks of different cutting points are different. The connection marks are represented as ,like Figure 3 As shown, Figure 3 The figure is a schematic diagram of the connection of the cutting points. The connection mark can be used as the head and tail of the data fragment to obtain multiple data fragments, which is convenient for the subsequent combination of the data fragments.

[0064] Step S30, comparing the redundant data with the library data in the redundant library, and when the comparison is consistent, determining the index value of the redundant data.

[0065] It should be noted that the index value refers to the information used to describe the position of redundant data in the redundant database, and is used to locate the position code of redundant data. The redundant database is a collection of all data in the database, in which the data in the redundant database are different from each other except for the redundant data with too short data length, that is, only one copy of the data is stored, and the criterion for the data length to be too short is the length of the index value, refer to Figure 4 , Figure 4 Construct a flow chart for the redundant database. When the length of the index value is less than the length of the redundant data, the redundant data is stored in the redundant database. When the length of the index value is greater than the length of the redundant data, the redundant data is not stored in the redundant database.

[0066] In a specific implementation, the redundant data is compared with the library data in the redundant library. The values ​​for comparison are the hash values ​​corresponding to the redundant data and the library data in the redundant library respectively. Due to the characteristics of the hash value, the hash values ​​obtained will be the same only when the data are completely consistent. Therefore, when the redundant data and the data in the redundant library are compared in hash values ​​and a consistent result is obtained, the index value corresponding to the library data in the corresponding redundant library can be determined.

[0067] In a feasible implementation manner, the step of comparing the redundant data with the library data in the redundant library, and when the comparison is consistent, before determining the index value of the redundant data, further includes: Determine the node relationship of the library data according to the redundant inclusion relationship between the library data, generate a redundant index tree according to the node relationship, and generate a tree code according to the library data; Encoding the nodes on the redundant index tree to determine the leaf codes of the library data in the redundant index tree; The tree code and the leaf code are combined to obtain a library data code, and the library data code is filled into corresponding positions in the index bit positions to obtain an index value.

[0068] In a specific implementation, when redundant data is added to a redundant library, the redundant inclusion relationship can be determined according to the redundant inclusion relationship between the library data, referring to Figure 5 , Figure 5 It is a diagram of redundant inclusion. It can be understood that when a data length DataA is longer, part of this data will appear in other data DataB. Therefore, it can be known that DataA contains DataB, and then it can be determined that the node relationship between DataA and DataB is that the former is the root node and the latter is the leaf node. Based on a similar method, the node relationship of the redundant library is obtained to generate a redundant index tree, and the tree code is generated according to the library data. The redundant tree is determined according to the type of redundant data. Different redundant trees correspond to different types of redundant data, that is, a redundant data can only exist in one redundant index tree. Refer to Figure 6 , Figure 6 The structure diagram of a redundant index tree. Each node includes a hash value of redundant data, and the node is encoded. Then the tree code is combined with the leaf code to obtain the library data code, and the library data code is filled into the corresponding position in the index bit to obtain the index value, for example, the library code data of node F is 11.

[0069] Step S40: insert the index value into the redundant position, merge and compress it with the independent data to obtain compressed data, and store the compressed data on a disk.

[0070] It should be noted that compressed data refers to data obtained by replacing part of the redundant data with the index value and combining it with the opposing data. Compared with the original data, the data size is reduced by a certain amount, and the reduced data amount is the difference between the redundant data and the index value data amount.

[0071] It can be understood that the process of inserting the index value into the redundant position and fusing and compressing it with the independent data to obtain the compressed data can be described as: determining the first character and the last character of the index value; sorting the independent data and the index value in the order of the data content of the newly added data to obtain a combined sequence; based on the combined sequence, connecting the first character of the index value with the last character of the independent data, and / or connecting the last character of the index value with the first character of the independent data to obtain fused data; fusing and compressing the fused data to obtain compressed data. Specifically, it is possible to determine the first character and the last character of the index value, sort the independent data and the index value in the order of the data content in the newly added data to obtain a combined sequence, then connect the last character of the index value with the first character of the independent data, or connect the last character of the index value with the first character of the independent data to obtain fused data, and compress the fused data to obtain fused data.

[0072] In a feasible implementation manner, the step of fusing and compressing the fused data to obtain compressed data includes: Determine an initial compressed segment, the initial compressed segment comprising a first compressed byte and a second compressed byte, the first compressed byte and the second compressed byte are at least one byte respectively; Determine an index value portion in the fused data, and separate the index value into a tree code and a leaf code; Fill the tree codes into the character bits in the first compressed byte respectively, and fill the leaf codes into the character bits in the second compressed byte respectively, to obtain compressed fragments; The compressed fragment replaces the index value part in the fused data to obtain compressed data.

[0073] It should be noted that the first compressed byte and the second compressed byte are two components of the index value. The byte length of the first compressed byte and the second compressed byte can be expanded according to the specific situation. In the initial state, it is 1 byte. The compressed fragment is the data fragment obtained after the compressed byte is filled with the index value.

[0074] In a specific implementation, the initial compressed segment consists of two compressed bytes, namely the first compressed byte and the second compressed byte. Each byte contains at least one byte of data. According to specific requirements and the characteristics of redundant data, a suitable byte length can be selected. In this embodiment, 8 bits are used as the length of the compressed byte. In the fused data, all inserted index values ​​are found. These index values ​​are identifiers used to replace redundant data before, and each index value is separated into two parts: a tree code (Tree Code) and a leaf code (Leaf Code). The tree code represents the structural information of the index value, and the leaf code represents the specific content of the index value. Then fill the tree code into the character bit in the first compressed byte, and fill the leaf code into the character bit in the second compressed byte. If the code is too long, multiple bytes are used to store it. After that, the first compressed byte and the second compressed byte filled with the tree code and the leaf code are combined into a compressed segment, and the corresponding index value part in the fused data is replaced with the generated compressed segment. In this way, the amount of data actually stored can be reduced while retaining the data structure. All replaced fused data segments are merged to form the final compressed data, and the compressed data is stored in a disk or other persistent storage medium. At the same time, necessary metadata (such as index mapping table, compression algorithm type, etc.) is saved for subsequent decompression and recovery.

[0075] In a feasible implementation manner, after the step of inserting the index value into the redundant position, fusing and compressing it with the independent data to obtain compressed data, and storing the compressed data on a disk, the step further includes: When a preset update time node is reached, the compressed data in the target area of ​​the disk is traversed, and independent data in the compressed data is extracted to obtain an independent data set; Performing similarity detection on the independent data set to determine duplicate data in the independent data set; The duplication degree of the duplicate data is determined, and when the duplication degree is greater than a redundancy threshold, the duplicate data is marked as new redundant data, and the new redundant data is added to the redundant library.

[0076] In a specific implementation, the database can periodically perform redundancy detection on the data in the database. When the preset update time node is reached, such as 03:00 on the 1st of each month, the compressed data in the target area of ​​the disk can be traversed, and the independent data therein can be extracted to obtain an independent data set. Then, similarity detection is performed on the data in the independent data set to determine the duplicate data in the independent data set, and the degree of duplication is determined based on the duplicate data. When the degree of duplication is greater than the redundancy threshold, the duplicate data can be marked as new redundant data, and the new redundant data is added to the redundant library, and the compressed data can be further compressed to reduce storage space occupancy.

[0077] The present embodiment provides a data storage method, which performs redundancy detection on newly added data to determine the redundant position of the newly added data, and divides the newly added data based on the redundant position to obtain redundant data and independent data; compares the redundant data with the library data in the redundant library, and when the comparison is consistent, determines the index value of the redundant data; inserts the index value into the redundant position, and merges and compresses it with the independent data to obtain compressed data, and stores the compressed data on a disk, so as to replace the data in the newly added data that is repeated in the database, and replaces the repeated part of the newly added data with the index value corresponding to the repeated data, so as to store the repeated data once and reference it multiple times, thereby reducing the storage resource occupancy, improving the utilization rate of the storage resources, and storing more data.

[0078] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data storage method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0079] The present application also provides a data storage device, please refer to Figure 7 , the data storage device comprises: A redundancy detection module 10 is used to perform redundancy detection on the newly added data and determine the redundant position of the newly added data; A data segmentation module 20, configured to segment the newly added data based on the redundant position to obtain redundant data and independent data; A data comparison module 30, for comparing the redundant data with the library data in the redundant library, and determining the index value of the redundant data when the comparison is consistent; The data compression module 40 is used to insert the index value into the redundant position, merge and compress it with the independent data to obtain compressed data, and store the compressed data in a disk.

[0080] In a feasible implementation manner, the redundancy detection module 10 is also used to perform block processing and cleaning processing on the newly added data to obtain preprocessed data; determine the data features of the preprocessed data, and perform hash operation on the data features to determine the newly added hash value of the newly added data; compare the redundant hash value with the newly added hash value one by one, and when the newly added hash value is consistent with the redundant hash value, determine the redundant position of the newly added data, and the redundant hash value is the hash value obtained by hash operation of the data in the redundant library.

[0081] In a feasible implementation manner, the redundancy detection module 10 is further used to compare the newly added hash value with the redundant hash value one by one, and when the newly added hash value is consistent with the redundant hash value, determine the redundant data corresponding to the redundant hash value; calculate the starting character and the ending character of the redundant data corresponding to the redundant hash value, and obtain the edit distance according to the starting character and the ending character; determine the repetition scale according to the edit distance, overlap the first position of the repetition scale with the first character of the newly added data, determine the coverage data, and when the similarity between the coverage data and the redundant data corresponding to the redundant hash value reaches a preset similarity threshold, record the current coverage interval; mark the starting position and the ending position of the coverage interval as redundant positions.

[0082] In a feasible implementation manner, the data segmentation module 20 is also used to determine the position identifier of the redundant position, determine the starting position and the ending position of the position identifier; determine the starting position and the ending position as cutting points, generate connection marks for the characters before and after the cutting points, and the connection marks of different cutting points are different; use the connection marks as the data fragment header or the data fragment tail, respectively, to obtain multiple data fragments; determine the data fragment between the starting position and the ending position as redundant data; determine the data fragment before the starting position and the data fragment after the ending position as independent data.

[0083] In a feasible implementation, the data comparison module 30 is also used to determine the node relationship of the library data based on the redundant inclusion relationship between the library data, generate a redundant index tree based on the node relationship, and generate a tree code based on the library data; encode the nodes on the redundant index tree to determine the leaf code of the library data in the redundant index tree; combine the tree code with the leaf code to obtain the library data code, and fill the library data code into the corresponding positions in the index bit positions to obtain the index value.

[0084] In a feasible implementation, the data compression module 40 is also used to determine the first character and the last character of the index value; sort the independent data and the index value according to the data content order of the newly added data to obtain a combined sequence; based on the combined sequence, connect the first character of the index value with the last character of the independent data, and / or connect the last character of the index value with the first character of the independent data to obtain fused data; and perform fusion compression on the fused data to obtain compressed data.

[0085] In a feasible implementation manner, the data compression module 40 is also used to determine an initial compression fragment, the initial compression fragment includes a first compression byte and a second compression byte, the first compression byte and the second compression byte are at least one byte respectively; determine the index value part in the fused data, separate the index value to obtain a tree code and a leaf code; fill the tree code into the character bits in the first compression byte respectively, and fill the leaf code into the character bits in the second compression byte respectively, to obtain a compression fragment; replace the index value part in the fused data with the compression fragment to obtain compressed data; inserting the index value into the redundant position, and fusing and compressing it with the independent data to obtain compressed data, and storing the compressed data on the disk also includes: when reaching a preset update time node, traversing the compressed data in the target area of ​​the disk, and extracting independent data from the compressed data to obtain an independent data set; performing similarity detection on the independent data set to determine duplicate data in the independent data set; determining the degree of duplication of the duplicate data, and when the degree of duplication is greater than a redundancy threshold, marking the duplicate data as new redundant data, and adding the new redundant data to the redundant library.

[0086] The data storage device provided by the present application adopts the data storage method in the above embodiment, which can solve the technical problem that a large amount of duplicate data occupies storage resources and causes waste of storage resources in data storage. Compared with the prior art, the beneficial effects of the data storage device provided by the present application are the same as the beneficial effects of the data storage method provided by the above embodiment, and other technical features in the data storage device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0087] The present application provides a data storage device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data storage method in the above-mentioned embodiment one.

[0088] Reference below Figure 8, which shows a schematic diagram of the structure of a data storage device suitable for implementing an embodiment of the present application. The data storage device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The data storage device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0089] like Figure 8 As shown, the data storage device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data storage device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. Communication device 1009 can allow data storage device to communicate with other devices wirelessly or wired to exchange data. Although data storage device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0090] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0091] The data storage device provided by the present application adopts the data storage method in the above embodiment to solve the technical problem of data storage. Compared with the prior art, the beneficial effects of the data storage device provided by the present application are the same as the beneficial effects of the data storage method provided by the above embodiment, and the other technical features in the data storage device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0092] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0093] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0094] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data storage method in the above-mentioned embodiment.

[0095] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0096] The computer-readable storage medium may be included in a data storage device; or may exist independently without being assembled into a data storage device.

[0097] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data storage device, the data storage device enables the following: to perform redundancy detection on the newly added data to determine the redundant position of the newly added data; to segment the newly added data based on the redundant position to obtain redundant data and independent data; to compare the redundant data with the library data in the redundant library, and when the comparison is consistent, to determine the index value of the redundant data; to insert the index value into the redundant position, and to merge and compress it with the independent data to obtain compressed data, and to store the compressed data on the disk.

[0098] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0099] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0101] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data storage method, and can solve the technical problem of data storage. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data storage method provided in the above-mentioned embodiment, and will not be elaborated here.

[0102] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data storage method when the computer program is executed by a processor.

[0103] The computer program product provided by the present application can solve the technical problem of data storage. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data storage method provided by the above embodiment, which will not be repeated here.

[0104] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data storage method, characterized in that: The data storage method comprises: Performing redundancy detection on the newly added data to determine the redundant position of the newly added data; Segmenting the newly added data based on the redundant position to obtain redundant data and independent data; Comparing the redundant data with the library data in the redundant library, and when the comparison is consistent, determining the index value of the redundant data; The index value is inserted into the redundant position, and is fused and compressed with the independent data to obtain compressed data, and the compressed data is stored in a disk.

2. The method according to claim 1, characterized in that The step of performing redundancy detection on the newly added data to determine the redundant position of the newly added data includes: The newly added data is divided into blocks and cleaned to obtain pre-processed data; Determining data features of the preprocessed data, and performing a hash operation on the data features to determine a newly added hash value of the newly added data; The redundant hash value is compared with the newly added hash value one by one, and when the newly added hash value is consistent with the redundant hash value, the redundant position of the newly added data is determined, and the redundant hash value is a hash value obtained by hashing the data in the redundant library.

3. The method according to claim 2, characterized in that The step of comparing the newly added hash value with the redundant hash value one by one and determining the redundant position of the newly added data when the newly added hash value is consistent with the redundant hash value comprises: Compare the newly added hash value with the redundant hash value one by one, and when the newly added hash value is consistent with the redundant hash value, determine the redundant data corresponding to the redundant hash value; Calculate the starting character and the ending character of the redundant data corresponding to the redundant hash value, and obtain the edit distance according to the starting character and the ending character; Determine a repetition scale according to the edit distance, overlap the first position of the repetition scale with the first character of the newly added data, determine the coverage data, and record the current coverage interval when the similarity between the coverage data and the redundant data corresponding to the redundant hash value reaches a preset similarity threshold; The start position and the end position of the coverage interval are marked as redundant positions.

4. The method according to claim 1, characterized in that The step of dividing the newly added data based on the redundant position to obtain redundant data and independent data includes: Determine a position identifier of the redundant position, and determine a starting position and an ending position of the position identifier; The starting position and the ending position are determined as cutting points, and characters before and after the cutting points are used to generate connection marks, wherein the connection marks of different cutting points are different from each other; The connection mark is used as a data segment header or a data segment tail to obtain multiple data segments, and the newly added data is divided into multiple data segments by the cutting point; Determine the data segment between the starting position and the ending position as redundant data; The data segment before the start position and the data segment after the end position are determined as independent data.

5. The method according to claim 1, characterized in that The step of comparing the redundant data with the library data in the redundant library, and when the comparison is consistent, before determining the index value of the redundant data, further includes: Determine the node relationship of the library data according to the redundant inclusion relationship between the library data, generate a redundant index tree according to the node relationship, and generate a tree code according to the library data; Encoding the nodes on the redundant index tree to determine the leaf codes of the library data in the redundant index tree; The tree code and the leaf code are combined to obtain a library data code, and the library data code is filled into corresponding positions in the index bit positions to obtain an index value.

6. The method according to claim 1, characterized in that The step of inserting the index value into the redundant position and fusing and compressing it with the independent data to obtain compressed data comprises: Determine the first and last characters of the index value; Sorting the independent data and the index values ​​according to the data content order of the newly added data to obtain a combined sequence; Based on the combination sequence, the first character of the index value is connected with the last character of the independent data, and / or the last character of the index value is connected with the first character of the independent data to obtain fused data; The fused data is fused and compressed to obtain compressed data.

7. The method according to claim 6, characterized in that The step of fusing and compressing the fused data to obtain compressed data comprises: Determine an initial compressed segment, the initial compressed segment comprising a first compressed byte and a second compressed byte, the first compressed byte and the second compressed byte are at least one byte respectively; Determine an index value portion in the fused data, and separate the index value into a tree code and a leaf code; Fill the tree codes into the character bits in the first compressed byte respectively, and fill the leaf codes into the character bits in the second compressed byte respectively, to obtain compressed fragments; Replacing the index value part in the fused data with the compressed fragment to obtain compressed data; After the step of inserting the index value into the redundant position and fusing and compressing it with the independent data to obtain compressed data, and storing the compressed data on a disk, the method further includes: When a preset update time node is reached, the compressed data in the target area of ​​the disk is traversed, and independent data in the compressed data is extracted to obtain an independent data set; Performing similarity detection on the independent data set to determine duplicate data in the independent data set; The duplication degree of the duplicate data is determined, and when the duplication degree is greater than a redundancy threshold, the duplicate data is marked as new redundant data, and the new redundant data is added to the redundant library.

8. A data storage device, characterized in that: The data storage device comprises: A redundancy detection module, used to perform redundancy detection on the newly added data and determine the redundant position of the newly added data; A data segmentation module, used for segmenting the newly added data based on the redundant position to obtain redundant data and independent data; A data comparison module, used to compare the redundant data with the library data in the redundant library, and when the comparison is consistent, determine the index value of the redundant data; The data compression module is used to insert the index value into the redundant position, merge and compress it with the independent data to obtain compressed data, and store the compressed data on a disk.

9. A data storage device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data storage method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 7 are implemented.

Images