Data management method, apparatus and device

Abstract

The application provides a data management method, device and equipment. The data management method comprises the following steps: taking a first row as an anchor point, storing the first row itself as a compression result, taking a second row as a target row, encoding the target row by using each encoding mode to obtain an encoding result, searching for a target encoding result with the minimum storage overhead, judging whether the target row meets any one of a plurality of specified conditions, storing the target row itself as the compression result if the target row meets any one of the plurality of specified conditions, otherwise storing the target encoding result as the compression result, taking a next row as the target row, and repeatedly performing the step of encoding the target row by using each encoding mode until the last row is processed, and combining compression results of all rows into a compression file and establishing index information of the compression file. The data management method, device and equipment provided by the application can improve the compression rate.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data management method, apparatus, and device. Background Technology

[0002] In recent years, with the rapid growth of data volume, the number of lines in log files has exceeded tens of millions or even hundreds of millions. This massive amount of data not only consumes a large amount of storage resources but also poses a significant challenge to data transmission. Currently, in order to improve data storage efficiency and transmission performance, data compression technology is often used to compress data, effectively removing redundant information and significantly reducing storage volume and improving transmission efficiency without compromising data integrity.

[0003] Although data compression technology has shown significant advantages in improving storage efficiency and transmission performance, the diversity and complexity of data types within the industry mean that the compression effect varies greatly for different types of data, often resulting in fluctuations in the compression ratio. Summary of the Invention

[0004] In view of this, this application provides a data management method, apparatus, and device to improve the data compression rate.

[0005] The first aspect of this application provides a data management method, the data management method comprising:

[0006] After using the first line of the data to be compressed as the anchor point and storing the first line itself as the compression result of the first line, the second line of the data to be compressed is used as the target line.

[0007] For each of the multiple encoding modes, the target line is encoded using that encoding mode to obtain the encoding result of the target line under that encoding mode;

[0008] Find the target encoding result with the minimum storage overhead from the encoding results of the target row under various encoding modes;

[0009] Determine whether the target row meets any one of a plurality of specified conditions; wherein the plurality of specified conditions include: the content of the target row undergoes a sudden change; the number of rows in the target data block containing the target row is greater than a first preset threshold; the storage overhead of the target data block is greater than a second preset threshold;

[0010] If so, the target line is used as the anchor point, and the target line itself is stored as the compression result of the target line; otherwise, the target encoding result is stored as the compression result of the target line.

[0011] Take the next line as the target line, and repeat the steps for each of the multiple encoding modes to encode the target line using that encoding mode until the last line is processed;

[0012] The compression results of all lines are combined into a compressed file of the data to be compressed, and index information is created for the compressed file.

[0013] A second aspect of this application provides a data management device, which includes a processing module, an encoding module, and a creation module;

[0014] The processing module is used to store the first line of the data to be compressed as the target line after using the first line as the anchor point and storing the first line itself as the compression result of the first line.

[0015] The encoding module is used to encode the target line using each of the multiple encoding modes to obtain the encoding result of the target line under that encoding mode.

[0016] The encoding module is also used to find the target encoding result with the minimum storage overhead from the encoding results of the target row under various encoding modes;

[0017] The encoding module is also used to determine whether the target row satisfies any one of a plurality of specified conditions;

[0018] The encoding module is further configured to, when the target line satisfies any one of the multiple specified conditions, use the target line as an anchor point and store the target line itself as the compression result of the target line; otherwise, store the target encoding result as the compression result of the target line.

[0019] The encoding module is further configured to take the next line as the target line, and execute the step of encoding the target line using each of the multiple encoding modes again, until the last line is processed;

[0020] The creation module is also used to combine the compression results of all rows into a compressed file of the data to be compressed, and to establish index information for the compressed file.

[0021] A third aspect of this application provides a data management device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described in the first aspect of this application.

[0022] A fourth aspect of this application provides a computer program product including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods provided in the first aspect of this application.

[0023] The data management method, apparatus, and device provided in this application employ multiple encoding modes for parallel encoding of each target line, and automatically select the encoding result with the optimal compression effect based on the storage overhead of each mode, thereby achieving an adaptive optimal compression strategy. Simultaneously, by setting line number thresholds, block size thresholds, and a content mutation detection mechanism, the compression blocks are dynamically divided and anchor points are updated in real time, effectively avoiding the problem of decreased compression efficiency caused by data mutations or excessively large data within blocks. This method, while ensuring compression efficiency, fully exploits the redundancy characteristics of inter-line data, significantly improving the overall compression ratio and maintaining the stability and consistency of the compression effect.

[0024] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 A flowchart of an embodiment of the data management method provided in this application;

[0027] Figure 2 A flowchart of Embodiment 2 of the data management method provided in this application;

[0028] Figure 3 A flowchart of Embodiment 3 of the data management method provided in this application;

[0029] Figure 4 A flowchart of Embodiment 4 of the data management method provided in this application;

[0030] Figure 5 A flowchart of Embodiment 5 of the data management method provided in this application;

[0031] Figure 6 This is a hardware structure diagram of a data management device, which is the data management apparatus, shown as an exemplary embodiment of this application.

[0032] Figure 7 A schematic diagram of the structure of the data management device according to Embodiment 1 of this application;

[0033] Figure 8This is a schematic diagram of the structure of Embodiment 2 of the data management device provided in this application. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0035] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0037] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0038] Figure 1 A flowchart illustrating an embodiment of the data management method provided in this application. Please refer to... Figure 1 The data management method provided in this embodiment may include:

[0039] S101. After using the first line of the data to be compressed as the anchor point and storing the first line itself as the compression result of the first line, the second line of the data to be compressed is used as the target line.

[0040] The data management method and apparatus provided in this application are applied to a data management device that supports row storage and organizes data by row.

[0041] Specifically, the data to be compressed can be data with a row structure. For example, the data to be compressed can be log data. Table 1 shows the data to be compressed in an exemplary embodiment of this application, which is server performance log data:

[0042] Table 1. Data to be compressed as illustrated in an exemplary embodiment of this application.

[0043]

[0044] It should be noted that, in block differential compression, the anchor point refers to the starting line of each compressed block, which is stored in its uncompressed, complete form and serves as the reference for reconstructing subsequent lines within that compressed block.

[0045] In this embodiment, the first line is used as the anchor point, and the first line itself is used as the compressed result of the first line for storage. Combining the example above, the compressed result stored at this time is as follows: [Anchor] 1665510000,svr-db-01,5.2,25.4,OK (approximately 38 bytes); where [Anchor] is used to indicate that the line is the anchor point.

[0046] Furthermore, after using the first row of data to be compressed as the anchor point and storing the first row itself as the compression result, the second row of data to be compressed is used as the target row.

[0047] S102. For each of the multiple encoding modes, the target line is encoded using that encoding mode to obtain the encoding result of the target line under that encoding mode.

[0048] The number and type of various encoding modes are set according to actual needs, and are not limited in this embodiment. For example, in one possible implementation, the various encoding modes include at least two of the following encoding modes: standard incremental encoding, cross-line reference incremental encoding, compressed differential, and compressed complete data.

[0049] Standard incremental encoding refers to encoding the changed parts based on the content difference between the target line and the previous line, recording only the difference information and not storing the same content repeatedly. Cross-line reference incremental encoding, based on standard incremental encoding, allows the current line to reference historical lines during the encoding process, thus achieving cross-line reuse in incremental compression. Compressed differential encoding refers to compressing the difference between the target line and the reference line again according to a specified compression algorithm. Compressed complete data refers to compressing complete data according to a specified compression algorithm. It should be noted that the specific implementation principles and processes of each encoding mode can be found in related technical descriptions, and will not be repeated here. Furthermore, the specified compression algorithm is set according to actual needs, and is not limited in this embodiment. For example, in one possible implementation, the specified compression algorithm could be LZ, Huffman, Zstd, etc.

[0050] In this step, for each encoding pattern, the target row is encoded using that pattern to obtain the encoding result of the target row under that encoding pattern. For example, in one possible implementation, multiple encoding patterns include: encoding pattern A (standard incremental encoding), encoding pattern B (inter-line reference incremental encoding), encoding pattern C (compressed differential), and encoding pattern D (compressed complete data). In this step, the target row is encoded using encoding pattern A to obtain encoding result a; similarly, the target row is encoded using encoding patterns B, C, and D respectively to obtain encoding results b, c, and d.

[0051] S103. Find the target encoding result with the minimum storage overhead from the encoding results of the target row under various encoding modes.

[0052] It should be noted that the storage overhead of the encoding result refers to the amount of storage space occupied by the encoding result, usually expressed in bytes.

[0053] In this step, the storage overhead of each encoding result is calculated, and then the target encoding result with the lowest storage overhead is found. For example, when the target line is the second line, the storage overhead of encoding result 'a' corresponding to encoding pattern A is 5 bytes, the storage overhead of encoding result 'b' corresponding to encoding pattern B is 5 bytes, the storage overhead of encoding result 'c' corresponding to encoding pattern C is much greater than 5 bytes, and the storage overhead of encoding result 'd' corresponding to encoding pattern D is 20 bytes. In this case, the target encoding result is determined to be encoding result 'a'.

[0054] S104. Determine whether the target row satisfies any one of the multiple specified conditions.

[0055] Specifically, the multiple specified conditions include: the content of the target row undergoes a sudden change; the number of rows in the current compressed block is greater than a first preset threshold; and the storage overhead of the current compressed block is greater than a second preset threshold.

[0056] In practice, the similarity between the target line and historical lines within the current compressed block can be used to determine whether the content of the target line has mutated. If the similarity between the target line and historical lines is low, it is determined that the content of the target line has mutated; otherwise, it is determined that the content of the target line has not mutated. For example, if the target line is the fourth line, then based on the similarity between the target line and historical lines within the current compressed block, it is determined that the content of the fourth line has mutated.

[0057] Optionally, in one possible implementation, the process of determining whether the content of the target line has undergone a sudden change may include:

[0058] (1) Determine whether the storage overhead of the target encoding result is greater than a third preset threshold; the third preset threshold is related to the original storage overhead of the target row.

[0059] (2) When the storage overhead of the target encoding result is greater than the third preset threshold, it is determined that the content of the target line has changed mutally; otherwise, it is determined that the content of the target line has not changed mutally.

[0060] Specifically, the specific value of the third preset threshold is set according to actual needs, and is not limited in this embodiment. For example, in one possible implementation, the third preset threshold = 0.7. Original storage overhead. Referring back to the previous example, for the second line, the storage overhead in encoding mode A is 5 bytes, which is no greater than the third preset threshold (at this point, the third preset threshold is...). This confirms that the content of the second line has not undergone a mutation.

[0061] Furthermore, the first and second preset thresholds are set according to actual needs. In this embodiment, the specific values ​​of the first and second preset thresholds are not limited. For example, in one possible implementation, the first preset threshold is 256, that is, when the number of rows in the current compressed block is greater than 256, the compressed block is considered to have ended, and the target row is used as the anchor point to create a new compressed block. As another possible implementation, the second preset threshold is 64KB, that is, when the storage overhead of the current compressed block is greater than 64KB, the compressed block is considered to have ended, and the target row is used as the anchor point to create a new compressed block.

[0062] It should be noted that the method provided in this embodiment comprehensively considers multiple dimensions such as changes in row content, the number of rows in the compressed block, and storage overhead. It can dynamically determine the anchor point position based on data characteristics, rather than relying on fixed intervals for partitioning. Through this adaptive anchor point mechanism, it can flexibly handle row structure data of different types and fluctuation characteristics. When a sudden change in data content is detected, the compression reference link is cut off in time to avoid the problem of subsequent row compression rate decrease or encoding error accumulation caused by the sudden data. In this way, it can not only effectively improve the overall compression rate, but also significantly enhance the stability and robustness of the compression effect.

[0063] S105. If yes, use the target line as the anchor point and store the target line itself as the compression result of the target line; otherwise, store the target encoding result as the compression result of the target line.

[0064] Based on the above description, when a target line meets any of the specified conditions, the target line is used as the anchor point. In this case, the target line itself is stored as the compressed result. For example, if the target line is the fourth line, a new compressed block is created, using the fourth line as the anchor point. The stored compressed result is as follows: Block 2: [Anchor_R4] 1665510003,svr-web-03,85.1,60.1,CRITICAL: High CPULoad (approximately 61 bytes).

[0065] Combining the example above, if the target line is the second line, and the second line does not meet any of the specified conditions, then the compressed result stored is as follows: [Delta_A] (pos 9, val '1'), (pos 24, val '8') (approximately 5 bytes); where [Delta_A] indicates that the encoding method is encoding mode A.

[0066] As described above, when the target line is not used as an anchor point, the encoding result with the lowest storage overhead is selected from multiple encoding results as the compression result of the target line. In this way, it can be determined that the average incremental size of the entire compressed file is always less than or equal to the average incremental size corresponding to any single fixed encoding pattern, which can effectively improve the compression ratio.

[0067] S106. Take the next line as the target line, and repeat the steps of encoding the target line using each of the multiple encoding modes until the last line is processed.

[0068] Specifically, after processing the current line, the next line is taken as the target line, and the process returns to step S102. For example, if the target line is the second line, the third line is taken as the target line in this step, and the process returns to step S102. Further, in step S102, the third line is encoded using each encoding mode to obtain the encoding results of the third line under various encoding modes. Further, in step S103, the target encoding mode is determined to be encoding mode A. Subsequently, in step S104, after judgment, it is determined that the third line does not meet any of the specified conditions. In step S105, the encoding result under encoding mode A is stored as the compressed result of the third line. The stored compressed result is as follows: [Delta_A] (pos 9, val '2'), (pos 24, val '9'), (pos 29, val '5') (approximately 6 bytes). Then, the fourth line is processed, and the last line is processed directly.

[0069] For example, in one embodiment, after loop processing, the compression result of line 4 is as follows: Block 2: [Anchor_R4]1665510003,svr-web-03,85.1,60.1,CRITICAL: High CPU Load (approximately 61 bytes); the compression result of line 5 is as follows: [Delta_B] (ptr to R1), (pos 9, val '4')……, (approximately 8 bytes); the compression result of line 6 is as follows: [Delta_C] Zstd(": AAAAA...") (approximately 10 bytes); where [Delta_B] indicates that the encoding mode used in line 5 is encoding mode B, (ptr to R1) indicates that the reference line is R1 under encoding mode B; [Delta_C] indicates that the encoding mode used in line 6 is encoding mode C.

[0070] S107. Combine the encoding results of all lines into a compressed file of the data to be compressed, and establish index information for the compressed file.

[0071] Specifically, in this step, the encoding results of all lines are combined to form a compressed file of the data to be compressed. Referring to the example above, the compressed file of the data to be compressed is:

[0072]

[0073] The total size of Block 1 is 49 bytes, and the total size of Block 2 is 79 bytes.

[0074] Furthermore, in one possible implementation, when creating index information for a compressed file, an index record corresponding to each anchor point can be created based on the line number of the anchor point and the physical offset corresponding to the anchor point, thus forming index information.

[0075] Based on the example above, the index information is as follows:

[0076] (Anchor_Row_Number: 1, Physical_Offset: 0);

[0077] (Anchor_Row_Number: 4, Physical_Offset: 49).

[0078] The method provided in this embodiment employs multiple encoding modes for parallel encoding of each target line, and automatically selects the encoding result with the optimal compression effect based on the storage overhead of each mode, thereby achieving an adaptive optimal compression strategy. Simultaneously, by setting line number thresholds, block size thresholds, and a content mutation detection mechanism, the method dynamically divides compression blocks and updates anchor points in real time, effectively avoiding the problem of decreased compression efficiency caused by data mutations or excessively large data within blocks. This method, while ensuring compression efficiency, fully exploits the redundancy characteristics of inter-line data, significantly improving the overall compression ratio and maintaining the stability and consistency of the compression effect.

[0079] Optionally, in one possible implementation, after finding the target encoding result with the minimum storage overhead from the encoding results of the target row under various encoding modes, the method includes:

[0080] The target line is added to the suffix automaton; wherein, the nodes in the suffix automaton represent the set of substrings that appear in the historical line, and the edges in the suffix automaton represent the new nodes to which a character is read from a node; each node also records the length of the longest substring corresponding to the node, the suffix link, the set of the end positions of the substring corresponding to the node in the historical line, and the number of times the substring corresponding to the node appears.

[0081] Specifically, the suffix automaton represents all substrings of a string with the fewest states and transitions. Nodes represent the equivalence classes of all substrings with the same right-hand side set. Edges represent the transitions to form a new substring after appending a character to the end of the current substring. Each node also records the length of the longest substring (max_length), the suffix link, the set of end positions of the substring corresponding to the node in the history (end_positions), and the occurrence count of the substring corresponding to the node (occurrence_count). The suffix link is used to quickly locate the node corresponding to the longest proper suffix of the current substring, thus ensuring the linear complexity of string matching and expansion operations.

[0082] In practice, the string content of the target line can be used as input, and the suffix automaton can be read character by character starting from its initial state. For each newly read character, if a transition for that character exists in the current state, the state is moved along that transition to the next state. If no corresponding transition exists, a new state node is created, and its max_length, suffix link, end_positions, and occurrence_count are updated. If a substring in the target line has already appeared in a historical line, the existing state is reused, and only the end_positions set of that state is updated. The above process is repeated until the entire line of characters has been read, thereby achieving online incremental updates to the suffix automaton.

[0083] It should be noted that the suffix automaton can dynamically maintain the index structure of all substrings of the historical row data, so that the similarity search and differential reference matching for the new row can be completed in O(L) time complexity (where L is the length of the target row), which significantly improves the search efficiency.

[0084] In one possible implementation, the multiple encoding modes include cross-line reference incremental encoding; the process of encoding the target line using cross-line reference incremental encoding includes: using the suffix automaton to search for the reference line that best matches the target line; using the reference line as a reference, and performing differential encoding on the target line according to the difference between the target line and the reference line.

[0085] Specifically, as described above, the core idea of ​​cross-line reference incremental encoding is: for the current target line to be compressed... Instead of relying solely on its physical preceding row, it uses a suffix automaton to search for the most similar row among all historical rows as a reference row and uses that reference row as the compression benchmark.

[0086] In specific implementation, the target line is read. The string content is scanned character by character from the initial state of the suffix automaton. If a transition edge exists, the process moves to the next state along that edge; if no transition edge exists, backtracking is performed using suffix links to find a shorter matching suffix and continuing the matching. Throughout the scan, the length of the longest matching substring and its last occurrence row number (last_occurrence_row_id) are recorded. After this search, the longest matching length (L_max) and the corresponding reference row ID can be determined, where reference row ID = last_occurrence_row_id(L_max).

[0087] The data management method provided in this embodiment constructs and maintains a suffix automaton. Utilizing the suffix automaton's state transition structure, it stores all substring information from historical rows in a compact form, enabling rapid matching and reference lookup of large-scale historical data. It can quickly retrieve the reference row most similar to the target row within the historical rows. Compared to traditional sequential comparison methods, this method effectively reduces time complexity and improves search efficiency. Furthermore, with improved search efficiency, the potential of cross-row reference incremental encoding is fully realized. This allows finding a reference row far superior to the physical preceding row within a larger historical window at a lower cost. Differential encoding based on this reference row can then significantly improve the compression ratio.

[0088] Figure 2 A flowchart illustrating Embodiment Two of the data management method provided in this application. Please refer to... Figure 2 Based on the above embodiments, the process of determining whether the content of the target row has undergone a sudden change may include:

[0089] S201. Determine the length of each substring contained in the target row and the frequency of each substring contained in the target row in the historical row according to the suffix automaton.

[0090] Referring to the preceding description, each node of the suffix automaton also records the length of the longest substring corresponding to that node and the frequency of occurrence of that substring. In this step, based on the suffix automaton, the lengths of each substring contained in the target row and the frequency of occurrence of each substring in the target row in the historical rows can be determined. Furthermore, based on the frequency of occurrence of each substring in the target row in the historical rows and the frequency of occurrence of all substrings recorded in the suffix automaton in the historical rows, the frequency of occurrence of each substring in the target row in the historical rows can be calculated.

[0091] S202. Determine the future revenue of the target row based on the length of each substring contained in the target row and the frequency of each substring contained in the target row in the historical rows; the future revenue is used to characterize the potential value of the target row as a dictionary row.

[0092] In a specific implementation, in one possible approach, for each substring included in the target row, the sub-revenue of the substring can be determined based on its length and frequency of occurrence; and then the sum of all sub-revenues of all substrings included in the target row can be determined as the future revenue of the target row.

[0093] Optionally, in one possible implementation, the process of determining the sub-revenue of a substring based on its length and frequency of occurrence may include:

[0094] The sub-profit of this substring is determined using the following formula:

[0095] =length( ) ;

[0096] in, Let i be the sub-profit of the i-th substring;

[0097] length ) is the i-th substring Length;

[0098] For the i-th substring Frequency of occurrence.

[0099] As described above, it is understandable that when the future revenue of a target line is high, it means that the target line contains a large number of "long and common" substrings, and has great potential to be cited in the future.

[0100] S203. When the future income is greater than or equal to the preset income, determine that the content of the target row has changed mutally; otherwise, determine that the content of the target row has not changed mutally.

[0101] It should be noted that the specific value of the preset revenue is set according to actual needs, and is not limited in this embodiment.

[0102] When the future revenue of a target row is greater than or equal to the preset revenue, it means that its future citation potential is huge. At this time, it is determined that the content of the target row has changed and it is used as the anchor point; otherwise, it is determined that the content of the target row has not changed.

[0103] The method provided in this embodiment uses a suffix automaton to count the length and frequency of substrings contained in the target line, and then determines the future revenue of the target line based on this. The potential value of the target line as a dictionary line is evaluated by the future revenue. In this way, the suffix automaton can accurately identify content mutations, update anchor points in a timely manner, prevent mutated data from having a negative impact on subsequent compression, and thus effectively maintain the stability and consistency of the compression ratio.

[0104] Figure 3 A flowchart illustrating Embodiment 3 of the data management method provided in this application. Please refer to... Figure 3 Based on the above embodiments, the process of establishing index information for the compressed file may include:

[0105] S301. For each anchor point, create a first-level index record corresponding to the anchor point based on the row number of the anchor point and the physical offset of the anchor point, thus forming the first-level index.

[0106] As described above, the compressed file contains several anchor points, each representing the starting position of a compressed block and corresponding to a directly locatable key location. In this step, for each anchor point, the following information is recorded: line number; physical offset; where the line number is the sequence number of the line corresponding to the anchor point in the original data, and the physical offset is the specific byte position of the anchor point in the compressed file. This information is then used to form a first-level index record, and the first-level index records of all anchor points are combined to obtain the first-level index.

[0107] S302. When the number of anchor points is less than or equal to the fourth preset threshold, the first-level index is determined as the index information.

[0108] It should be noted that the specific value of the fourth preset threshold is set according to actual needs, and is not limited in this embodiment. For example, in one possible implementation, the fourth preset threshold is 100.

[0109] Specifically, when the number of anchor points is no greater than the fourth preset threshold, only the first-level index is needed, and the second-level index is not required.

[0110] S303. When the number of anchor points is greater than the fourth preset threshold, the first anchor point is selected as the anchor point. Based on the row number of the selected anchor point and the physical offset of the selected anchor point recorded in the first-level index, a second-level index record corresponding to the selected anchor point is established.

[0111] S304. Select the next anchor point as the selected anchor point at a preset interval, and then execute the step of establishing the second-level index record corresponding to the selected anchor point based on the row number of the selected anchor point and the physical offset of the selected anchor point recorded in the first-level index to form the second-level index.

[0112] Specifically, when the number of anchor points exceeds the fourth preset threshold, the first-level index becomes too large, reducing query efficiency, and a second-level index needs to be created. Specifically, the second-level index can be formed by sampling a subset of anchor points at intervals to speed up the search process.

[0113] In specific implementation, in steps S303 and S304, starting from the first anchor point, the first entry of the second-level index is selected. The row number of this anchor point and its corresponding physical offset in the first-level index are recorded to form a second-level index record. The next anchor point is selected as a second-level index entry at a preset interval, and this process is repeated until the last anchor point. Through interval sampling, the second-level index only saves the reference information of a portion of the anchor points, thus forming a sparse index.

[0114] It should be noted that the specific value of the preset interval is set according to actual needs, and is not limited in this embodiment. For example, in one possible implementation, the preset interval can be determined based on the number of anchor points. The preset interval can be equal to the number of anchor points / k; where k is the number of secondary index records expected to be included in the secondary index.

[0115] Combining the example above, for instance, in one possible implementation, the second-level index is created as follows: (Anchor_Row_Number: 1, MainIndexOffset: 0).

[0116] S305. Combine the second-level index and the first-level index to form the index information.

[0117] In practice, this step combines and stores the first-level index and the second-level index to form complete index information.

[0118] The method provided in this embodiment establishes a first-level index record for each anchor point, which can accurately record the line number and physical offset of each anchor point in the compressed file, enabling rapid location. When the number of anchor points is large, a second-level index is introduced. In this way, when searching for data, the position near the target anchor point can be quickly located in the second-level index first, and then the search can be performed precisely in the first-level index, which greatly reduces the search overhead and improves the search efficiency.

[0119] Figure 4 The flowchart for Embodiment 4 of the data management method provided in this application is shown below. Please refer to... Figure 4 The method provided in this embodiment, based on the above embodiments, further includes:

[0120] S401. In response to the read request, obtain the read row number indicated by the read request.

[0121] Specifically, the read request can be a user-initiated read request that carries the read line number. In this step, the read line number is directly obtained from the read request. Referring to the example above, for instance, in one possible implementation, the read line number is R5.

[0122] S402. When the index information does not have a second-level index, search for the first matching row that matches the read row number from the first-level index; the first matching row is the row number in the first-level index that is before the read row number and closest to the read row number.

[0123] S403. When the index information has a second-level index, search for the first target row number that matches the read row number from the second-level index; the first target row number is the row number in the second-level index that is before the read row number and closest to the read row number.

[0124] Specifically, the index information may only exist in the first-level index, or it may exist in both the first-level and second-level indexes. When the index information only exists in the first-level index, in step S402, a linear search is performed directly in the first-level index. In practice, the first-level index records in the first-level index can be traversed to find the largest row number that is less than or equal to the row number being read. The found largest row number is the first matching row number. The physical offset position corresponding to this largest row number can be used as the decompression starting point, and reconstruction begins from this position.

[0125] Similarly, when both first-level and second-level indexes exist, the search is first performed in the second-level index, and then in the first-level index to improve efficiency. In practice, a linear traversal of the second-level index records can be used to find the largest row number less than or equal to the row number being searched; this found row number is the first target row number.

[0126] In a specific implementation, in one possible approach, the process of searching for the first matching row with the read row number from the first-level index, or the process of searching for the first target row with the read row number from the second-level index, may include:

[0127] (1) Take the first-level index or the second-level index as the object to be searched, and divide the object to be searched into the first interval and the second interval according to the dichotomy method.

[0128] In practice, the index currently used for location (first-level index or second-level index) is taken as the object to be searched, and it is divided into two intervals of equal or approximately equal length using a bisection method. For example, the object to be searched is divided into two intervals by the median of all row numbers contained in the object, resulting in the first interval and the second interval.

[0129] It is understandable that each interval includes several index records, and each index record corresponds to a row number and its physical offset.

[0130] (2) Find the target interval where the read row number is located from the first interval and the second interval.

[0131] Specifically, the read row number is compared with the first interval and the second interval respectively to determine the target interval where the read row number is located. It can be understood that when the read row number is greater than or equal to the row number corresponding to the start point of a certain interval, and less than or equal to the row number corresponding to the end point of that interval, that interval is the target interval where the read row number is located. For example, in one embodiment, the target interval where the read interval is located is the first interval.

[0132] (3) When the length of the target interval is less than or equal to the fifth preset threshold, find the row that matches the read row number in the target interval.

[0133] It should be noted that the specific value of the fifth preset threshold is set according to actual needs, and is not limited in this embodiment. For example, in one possible implementation, the fifth preset threshold is 16.

[0134] When the length of the target interval is less than or equal to 16, it is considered that the range of row numbers contained in the target interval is small. Therefore, a precise search is performed directly within the target interval, and the physical offset corresponding to the read row number is quickly located by matching the row numbers.

[0135] (4) When the length of the target interval is greater than the fifth preset threshold, the target interval is taken as the object to be searched, and the step of dividing the object to be searched into the first interval and the second interval according to the bisection method is executed again until the length of the target interval is less than the fifth preset threshold.

[0136] Combining the above example, for instance, when the length of the first interval is greater than the fifth preset threshold, it is considered that the amount of data in the first interval is still large. At this time, the first interval is taken as the object to be searched, and the first interval is divided into two intervals again according to the bisection method. Then, the target interval where the row number is located is searched and read from these two intervals again, until the length of the target interval is less than the fifth preset threshold, and then the row number is directly searched and read from the target interval.

[0137] The method provided in this embodiment recursively divides the object to be searched into smaller intervals and switches to linear search when the interval length is less than a fifth preset threshold. In this way, the search complexity can be reduced from linear to logarithmic, which significantly improves the overall efficiency of index search.

[0138] S404. Search for the specified neighborhood corresponding to the first target row number in the first-level index, and search for the second matching row that matches the read row number in the specified neighborhood.

[0139] Specifically, the starting point of the specified neighborhood is the first target row number, and the distance between the ending point of the specified neighborhood and the first target row number is equal to the preset interval; the second target row number is the row number within the specified neighborhood that is before the read row number and is closest to the read row number.

[0140] In practice, based on the first target row number obtained in the previous step, a specified neighborhood in the first-level index can be determined in this step. The starting point of the specified neighborhood is the first target row number, and the ending point is the anchor point whose row number difference is equal to a preset interval.

[0141] Furthermore, searching within this specified neighborhood, rather than traversing the entire first-level index, can significantly narrow the search scope.

[0142] S405. Reconstruct the data corresponding to the read row number based on the index record corresponding to the first matching row or the second matching row.

[0143] Specifically, once a matching row is found (either the first or second matching row), its index record contains the corresponding physical offset. In this step, the compressed result can be read from this physical offset and decoded line by line until the original data corresponding to the read row number is restored.

[0144] For example, following the example above, in one possible implementation, line number 5 is read, and the following index record is searched in the first-level index: (Anchor_Row_Number: 4, Physical_Offset: 49). At this point, the system jumps directly to the 49th byte of the compressed file and reads the complete data of R4: 1665510003,svr-web-03,85.1,60.1,CRITICAL: High CPU Load. Further, the next compression result is read sequentially, revealing [Delta_B] (ptrto R1), diff_data(R5, R1).

[0145] At this point, the encoding mode is determined to be encoding mode B, and the reference line is R1. Since R1 is in another compressed block, R1 is read according to the pointer information, and its content is obtained as: 1665510000,svr-db-01,5.2,25.4,OK. Finally, diff_data(R5, R1) is applied to R1 to reconstruct the complete content of R5, which is as follows: 1665510004,svr-db-01,6.1,25.6,OK.

[0146] The method provided in this embodiment directly utilizes the first-level index to complete the location when the number of indexed records is small, avoiding unnecessary index traversal. When the number of indexed records is large, a coarse-grained location is first performed using the second-level index, followed by a precise search within the specified neighborhood of the first-level index, thereby effectively reducing the search range and retrieval time. By combining hierarchical indexing with neighborhood search, the read operation maintains low time complexity even with increased data volume, improving random access efficiency.

[0147] As described above, it can be understood that the method provided in this embodiment significantly improves the efficiency and intelligence of adaptive encoding mode selection and dynamic anchor placement by adding the target line to the suffix automaton. Furthermore, by establishing a secondary index, read performance can be guaranteed.

[0148] A more specific embodiment is given below to illustrate the specific solution of this application in detail. Specifically, Figure 5 The flowchart for Embodiment 5 of the data management method provided in this application is shown below. Please refer to... Figure 5 The data management method provided in this embodiment may include:

[0149] S501. Take the first line of the data to be compressed as the anchor point and store the first line itself as the compression result of the first line.

[0150] S502, Take the second row of the data to be compressed as the target row.

[0151] S503. For each of the multiple encoding modes, the target line is encoded using that encoding mode to obtain the encoding result of the target line under that encoding mode.

[0152] S504. Find the target encoding result with the minimum storage overhead from the encoding results of the target row under various encoding modes.

[0153] S505. Add the target line to the suffix automaton.

[0154] S506. Determine whether the number of rows in the current compressed block is greater than the first preset threshold. If not, proceed to step S507. If yes, proceed to step S512.

[0155] S507. Determine whether the storage overhead of the current compressed block is greater than the second preset threshold. If not, proceed to step S508. If yes, proceed to step S512.

[0156] S508. Determine whether the storage overhead of the target row in the target encoding mode is greater than a third preset threshold. If not, execute S509; if yes, execute S512.

[0157] S509. Determine the length of each substring contained in the target row and the frequency of each substring contained in the target row in the historical row according to the suffix automaton.

[0158] S510. Determine the future revenue of the target row based on the length of each substring contained in the target row and the frequency of each substring contained in the target row appearing in the historical rows.

[0159] S511. Determine whether the future income is greater than or equal to the preset income. If yes, execute S512; otherwise, execute S513.

[0160] S512. Use the target line as an anchor point and store the target line itself as the compression result of the target line.

[0161] S513. Store the target encoding result as the compression result of the target row.

[0162] S514. Take the next line as the target line, and repeat the steps of encoding the target line using each of the multiple encoding modes until the last line is processed.

[0163] S515. Combine the encoding results of all lines into a compressed file of the data to be compressed, and establish index information for the compressed file.

[0164] The specific implementation principles and processes of steps S501 to S515 can be found in the descriptions in related technologies, and will not be repeated here.

[0165] The method provided in this embodiment can dynamically adjust the compression method according to different storage costs and future benefit assessments, ensuring that the compression effect of each row of data reaches the best level, and can effectively improve compression efficiency.

[0166] Corresponding to the aforementioned embodiment of a data management method, this application also provides an embodiment of a data management device.

[0167] An embodiment of a data management device disclosed in this application can be applied to a data management device. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the data management device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 As shown, Figure 6 This is a hardware structure diagram of a data management device, which is the data management apparatus shown in an exemplary embodiment of this application. (Except for...) Figure 6In addition to the processor, memory, network interface, and non-volatile memory shown, the data management device in the embodiment may also include other hardware depending on the actual function of the data management device, which will not be described in detail here.

[0168] Figure 7 This is a schematic diagram of the structure of a data management device according to Embodiment 1 of this application. Please refer to... Figure 7 The data management device provided in this embodiment includes a processing module 710, an encoding module 720, and a creation module 730.

[0169] The processing module 710 is used to store the second line of the data to be compressed as the target line after using the first line of the data to be compressed as the anchor point and storing the first line itself as the compression result of the first line.

[0170] The encoding module 720 is used to encode the target line using each of the multiple encoding modes to obtain the encoding result of the target line under the encoding mode.

[0171] The encoding module 720 is also used to find the target encoding result with the minimum storage overhead from the encoding results of the target row under various encoding modes;

[0172] The encoding module 720 is also used to determine whether the target row satisfies any one of a plurality of specified conditions;

[0173] The encoding module 720 is further configured to, when the target line satisfies any one of the multiple specified conditions, use the target line as an anchor point and store the target line itself as the compression result of the target line; otherwise, store the target encoding result as the compression result of the target line.

[0174] The encoding module 720 is further configured to take the next line as the target line, and execute the step of encoding the target line using each of the multiple encoding modes again, until the last line is processed;

[0175] The creation module 730 is also used to combine the compression results of all rows into a compressed file of the data to be compressed, and to establish index information for the compressed file.

[0176] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0177] Optionally, in one possible implementation, the processing module 710 is further configured to add the target line to the suffix automaton after finding the target encoding result with the minimum storage overhead from the encoding results of the target line in various encoding modes;

[0178] In this suffix automaton, each node represents the set of substrings that appear in the history line, and each edge represents a new node that is moved to after reading a character from a node. Each node also records the length of the longest substring corresponding to the node, the suffix link, the set of ending positions of the substring corresponding to the node in the history line, and the number of times the substring corresponding to the node appears.

[0179] Optionally, in one possible implementation, the encoding module 720 is specifically used for:

[0180] Determine whether the storage overhead of the target row in the target encoding mode is greater than a third preset threshold; the third preset threshold is related to the original storage overhead of the target row;

[0181] When the storage overhead of the target line in the target encoding mode is greater than a third preset threshold, it is determined that the content of the target line has undergone a sudden change; otherwise, it is determined that the content of the target line has not undergone a sudden change.

[0182] And / or,

[0183] The suffix automaton is used to determine the length of each substring contained in the target line and the frequency of each substring contained in the target line in the historical lines.

[0184] The future revenue of the target row is determined based on the length of each substring contained in the target row and the frequency of each substring in the target row appearing in the historical rows; the future revenue is used to characterize the potential value of the target row as a dictionary row.

[0185] If the future return is greater than or equal to the preset return, it is determined that the content of the target row has undergone a sudden change; otherwise, it is determined that the content of the target row has not undergone a sudden change.

[0186] Optionally, in one possible implementation, the multiple encoding modes include interline reference incremental encoding; the process of encoding the target line using interline reference incremental encoding includes:

[0187] The suffix automaton is used to search for the reference row that best matches the target row;

[0188] The target row is differentially encoded based on the reference row and the difference between the target row and the reference row.

[0189] Optionally, in one possible implementation, the creation module 730 is further specifically used for:

[0190] For each anchor point, a first-level index record is created based on the row number of the anchor point and the physical offset of the anchor point, forming the first-level index;

[0191] When the number of anchor points is less than or equal to the fourth preset threshold, the first-level index is determined as the index information;

[0192] When the number of anchor points is greater than the fourth preset threshold, the first anchor point is selected as the anchor point. Based on the row number of the selected anchor point and the physical offset of the selected anchor point recorded in the first-level index, a second-level index record corresponding to the selected anchor point is established.

[0193] Select the next anchor point as the selected anchor point at a preset interval, and then execute the step of building the second-level index record corresponding to the selected anchor point based on the row number of the selected anchor point and the physical offset of the selected anchor point recorded in the first-level index to form the second-level index.

[0194] The second-level index and the first-level index are combined to form the index information.

[0195] Figure 8 This is a schematic diagram of Embodiment 2 of the data management device provided in this application. Please refer to... Figure 8 Based on the above embodiments, in one possible implementation, the data management device further includes a decoding module 740, which is specifically used for:

[0196] In response to a read request, obtain the read line number indicated by the read request;

[0197] If the index information does not have a second-level index, the first matching row that matches the read row number is searched from the first-level index; the first matching row is the row number in the first-level index that is before the read row number and closest to the read row number;

[0198] When a second-level index exists in the index information, a first target row number matching the read row number is searched from the second-level index; the first target row number is the row number in the second-level index that is before the read row number and closest to the read row number;

[0199] The first target row number is located in the first-level index, and a second matching row is located in the specified neighborhood, which is equal to the reading row number. The starting point of the specified neighborhood is the first target row number, and the distance between the ending point of the specified neighborhood and the first target row number is equal to the preset interval. The second matching row is the row number in the specified neighborhood that is before the reading row number and is closest to the reading row number.

[0200] Reconstruct the data corresponding to the read row number based on the index record corresponding to the first matching row or the second matching row.

[0201] Optionally, in one possible implementation, the decoding module 740 is further specifically used for:

[0202] Take the first-level index or the second-level index as the object to be searched, and divide the object to be searched into a first interval and a second interval according to the binary search method;

[0203] Find the target interval containing the read row number from the first interval and the second interval;

[0204] When the length of the target interval is less than or equal to a fifth preset threshold, a line matching the read line number is searched from the target interval;

[0205] When the length of the target interval is greater than the fifth preset threshold, the target interval is taken as the object to be searched, and the step of dividing the object to be searched into the first interval and the second interval according to the bisection method is performed again until the length of the target interval is less than the fifth preset threshold.

[0206] Please continue to refer to Figure 6 This application also provides a data management device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0207] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods provided in the first aspect of this application.

[0208] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0209] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0210] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data management method characterized by, The data management method comprises: after taking a first row of to-be-compressed data as an anchor point and storing the first row itself as a compression result of the first row, taking a second row of the to-be-compressed data as a target row; encoding the target row by using each of a plurality of encoding modes to obtain an encoding result of the target row in the encoding mode; finding a target encoding result with the least storage overhead from the encoding results of the target row in various encoding modes; judging whether the target row meets any of a plurality of specified conditions; wherein the plurality of specified conditions comprise: a content mutation of the target row; a number of rows of a current compression block being greater than a first preset threshold; a storage overhead of the current compression block being greater than a second preset threshold; if yes, storing the target row as an anchor point and storing the target row itself as a compression result of the target row, or storing the target encoding result as the compression result of the target row; taking a next row as a target row and performing again the step of encoding the target row by using each of a plurality of encoding modes to obtain an encoding result of the target row in the encoding mode until the last row is processed; combining the compression results of all rows into a compressed file of the to-be-compressed data and establishing index information for the compressed file.

2. The method of claim 1, wherein, After the step of finding a target encoding result with the least storage overhead from the encoding results of the target row in various encoding modes, the method comprises: adding the target row to a suffix automaton; wherein a node in the suffix automaton represents a substring set appearing in historical rows, an edge in the suffix automaton represents a new node to which a transition is made after reading in a character from a node, and each node further records a length of a longest substring corresponding to the node, a suffix link, a set of ending positions of the substring corresponding to the node in the historical rows, and a frequency of occurrence of the substring corresponding to the node.

3. The method of claim 2, wherein, The process of judging whether the content of the target row mutates comprises: judging whether a storage overhead of the target encoding result is greater than a third preset threshold; the third preset threshold is related to an original storage overhead of the target row; when the storage overhead of the target encoding result is greater than the third preset threshold, determining that the content of the target row mutates, otherwise determining that the content of the target row does not mutate; and / or determining lengths of each substring contained in the target row and frequencies of occurrence of each substring contained in the target row in historical rows according to the suffix automaton; determining a future benefit of the target row according to the lengths of each substring contained in the target row and the frequencies of occurrence of each substring contained in the target row in the historical rows; the future benefit is used to represent a potential value of the target row as a dictionary row; when the future benefit is greater than or equal to a preset benefit, determining that the content of the target row mutates, otherwise determining that the content of the target row does not mutate.

4. The method of claim 2, wherein, The plurality of encoding modes comprise cross-row reference delta encoding; the process of encoding the target row by using cross-row reference delta encoding comprises: searching for a reference row most matching the target row by using the suffix automaton; Differential encoding the target row according to a difference of the target row relative to the reference row.

5. The method of claim 1, wherein, The index information is established for the compressed file, including: For each anchor point, a first-level index record corresponding to the anchor point is established according to a line number where the anchor point is located and a physical offset corresponding to the anchor point, to form a first-level index; When the number of anchor points is less than or equal to a fourth preset threshold, the first-level index is determined as the index information; When the number of anchor points is greater than the fourth preset threshold, a first anchor point is selected as a selected anchor point, and a second-level index record corresponding to the selected anchor point is established according to a line number where the selected anchor point is located and a physical offset recorded in the first-level index for the selected anchor point; A next anchor point is selected as the selected anchor point at a preset interval, and the step of establishing the second-level index record according to the line number where the selected anchor point is located and the physical offset recorded in the first-level index for the selected anchor point is executed again to form a second-level index; The second-level index and the first-level index are combined to form the index information.

6. The method of claim 5, wherein, The method further includes: In response to a read request, a read line number indicated by the read request is obtained; When the index information does not have a second-level index, a first matching line matching the read line number is found from the first-level index; the first matching line is a line number before the read line number and closest to the read line number in the first-level index; When the index information has a second-level index, a first target line number matching the read line number is found from the second-level index; the first target line number is a line number before the read line number and closest to the read line number in the second-level index; A specified neighborhood corresponding to the first target line number is found from the first-level index, and a second matching line matching the read line number is found from the specified neighborhood; a start point of the specified neighborhood is the first target line number, and a distance between an end point of the specified neighborhood and the first target line number is equal to the preset interval; the second matching line is a line number before the read line number and closest to the read line number in the specified neighborhood; Data corresponding to the read line number is reconstructed according to an index record corresponding to the first matching line or the second matching line.

7. The method of claim 6, wherein, Finding the first matching line matching the read line number from the first-level index or finding the first target line number matching the read line number from the second-level index includes: The first-level index or the second-level index is taken as a to-be-found object, and the to-be-found object is divided into a first interval and a second interval according to a binary search method; A target interval where the read line number is located is found from the first interval and the second interval; When an interval length of the target interval is less than or equal to a fifth preset threshold, a line matching the read line number is found from the target interval; When the interval length of the target interval is greater than the fifth preset threshold, the target interval is taken as a to-be-searched object, and the step of dividing the to-be-searched object into a first interval and a second interval according to the dichotomy is executed again until the interval length of the target interval is less than the fifth preset threshold.

8. A data management apparatus characterized by comprising: The data management device comprises a processing module, an encoding module and a creating module; The processing module is configured to store a first row of the to-be-compressed data as an anchor point and the first row itself as a compression result of the first row, and then store a second row of the to-be-compressed data as a target row; The encoding module is configured to encode the target row by using each of a plurality of encoding modes to obtain an encoding result of the target row in the encoding mode; The encoding module is further configured to search for a target encoding result with the least storage overhead from the encoding results of the target row in the various encoding modes; The encoding module is further configured to determine whether the target row satisfies any of a plurality of specified conditions; The encoding module is further configured to store the target row itself as a compression result of the target row when the target row satisfies any of the specified conditions, or store the target encoding result as the compression result of the target row when the target row does not satisfy any of the specified conditions; The encoding module is further configured to take a next row as a target row and execute again the step of encoding the target row by using each of the plurality of encoding modes until the last row is processed. The creating module is further configured to combine the compression results of all the rows into a compressed file of the to-be-compressed data and establish index information for the compressed file.

9. A data management device, characterized by comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the program.

10. A computer program product, characterised in that, A computer program / instruction, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 7.

Images