Data compression method and device, electronic equipment and readable storage medium
By constructing a data dictionary and location index, the problems of high computational complexity and low efficiency of existing compression algorithms are solved, achieving more efficient data compression and storage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MIGU CO LTD
- Filing Date
- 2023-09-07
- Publication Date
- 2026-05-12
AI Technical Summary
Existing compression algorithms have high computational complexity and low compression efficiency, making it difficult to efficiently compress data storage.
By constructing a data dictionary and a location index, and utilizing the unique field values stored in the data dictionary for each field, and converting the field values of each data record into corresponding location indexes, compression complexity is reduced and compression efficiency is improved.
It reduces compression complexity, improves compression efficiency, saves storage space, and simplifies the calculation process.
Smart Images
Figure CN117194352B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing technology, specifically relating to a data compression method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] To save data storage space, data is typically compressed before storage. In existing technologies, databases usually employ dictionary-based compression algorithms, such as the LZ777 compression algorithm, to compress and store data, reducing the disk space occupied. The essence of dictionary-based compression algorithms is to continuously find common parts in the data and then replace these common parts with symbols to achieve compression. Because it requires constantly finding common parts and replacing them with symbols, the computational complexity of compression is high, resulting in low compression efficiency. Summary of the Invention
[0003] This application provides a data compression method, apparatus, electronic device, and readable storage medium, which can solve the problems of high computational complexity and low compression efficiency of existing compression algorithms.
[0004] In a first aspect, embodiments of this application provide a data compression method, including:
[0005] Retrieve N data records, each data record including multiple field values, each field value corresponding to a field;
[0006] A data dictionary is constructed based on the frequency of repetition of the field values corresponding to the same field in the N data records, wherein the data dictionary stores the unique field values for each field;
[0007] Determine the position index of each field value of each data record in the data dictionary, and convert each field value of each data record into the corresponding position index to obtain the position index record corresponding to each data record;
[0008] Store the data dictionary and the location index record corresponding to each data record.
[0009] Optionally, after obtaining N data records and before storing the data dictionary and the position index record corresponding to each data record, the method further includes:
[0010] A baseline data record is determined based on the frequency of repetition of each field value in the corresponding field of each data record. The baseline data record is the data record with the highest sum of the repetition frequencies of each field value among the N data records.
[0011] After determining the position index of each field value of each data record in the data dictionary, and before storing the data dictionary and the position index record corresponding to each data record, the method further includes:
[0012] An index area is constructed based on the difference information between the position index record corresponding to other data records and the position index record corresponding to the benchmark data record. The index area stores the position index record corresponding to the benchmark data record and the difference index record of the other data records relative to the benchmark data record. The other data records are the data records other than the benchmark data record among the N data records.
[0013] The storage of the data dictionary and the location index record corresponding to each data record includes:
[0014] Store the data dictionary and the index area.
[0015] Optionally, the field values corresponding to the same field in the N data records that are not repeated are stored in the same row of the data dictionary, and the field values of the same field stored in each row of the data dictionary are sorted according to the frequency of repetition; the field values of different fields in the N data records are stored in different rows of the data dictionary.
[0016] Determining the position index of each field value in the data dictionary for each data record includes:
[0017] Based on the corresponding row data of each field value in the data dictionary, the column number of each field value in the corresponding row data is determined as the position index.
[0018] Optionally, the field values of the same field stored in each row of the data dictionary are sorted in descending order of repetition frequency.
[0019] Optionally, determining the baseline data record based on the repetition frequency of each field value in the corresponding field of each data record includes:
[0020] Using each field value of each data record as a node, construct a trie corresponding to the N data records, and merge duplicate field values of the same field into the same node during the construction of the trie. The trie includes N paths, and each path corresponds to one data record among the N data records.
[0021] Obtain the repetition frequency of each node in each path, and determine the sum of the repetition frequencies of each node in each path;
[0022] The data record corresponding to the path with the highest sum of repetition frequencies of all nodes among the N paths is determined as the baseline data record.
[0023] Optionally, constructing the index area based on the difference information between the location index record corresponding to other data records and the location index record corresponding to the reference data record includes:
[0024] Identify the target field in the location index record corresponding to the first data record that differs from the location index record corresponding to the reference data record, and determine the location index difference corresponding to the target field, wherein the first data record is any one of the other data records;
[0025] The sequence number of the target field and the position index difference are associated and written into the index area.
[0026] Optionally, the step of associating the sequence number of the target field and the position index difference and writing it into the index area includes:
[0027] If multiple target fields in the first data record have the same position index difference, the multiple target fields are associated with one of the position index differences and written into the index area.
[0028] Secondly, this application also provides a data decompression method, including:
[0029] Obtain compressed data, wherein the compressed data is obtained by compressing N data records using the data compression method described in the first aspect, and the compressed data includes a data dictionary corresponding to the N data records and a position index record corresponding to each data record;
[0030] Based on the location index record corresponding to each data record, the value of each field of each data record is queried from the data dictionary to obtain each data record.
[0031] Optionally, when the compressed data includes a data dictionary and an index area, the step of querying the data dictionary for each field value of each data record based on the position index record corresponding to each data record to obtain each data record includes:
[0032] Based on the position index record corresponding to the baseline data record stored in the index area, and the difference index record of other data records relative to the baseline data record, the position index record corresponding to the other data record is obtained, wherein the other data record is the data record other than the baseline data record among the N data records;
[0033] Based on the location index record corresponding to the baseline data record and the location index records corresponding to the other data records, the baseline data record is obtained by querying each field value of the baseline data record from the data dictionary, and the other data records are obtained by querying each field value of each data record in the other data records from the data dictionary.
[0034] Thirdly, this application provides a data compression apparatus, comprising:
[0035] The data acquisition module is used to acquire N data records. Each data record includes multiple field values, and each field value corresponds to a field.
[0036] A dictionary construction module is used to construct a data dictionary based on the repetition frequency of field values corresponding to the same field in the N data records, wherein the data dictionary stores the unique field values for each field;
[0037] The location index module is used to determine the location index of each field value of each data record in the data dictionary, and convert each field value of each data record into the corresponding location index to obtain the location index record corresponding to each data record;
[0038] The record storage module is used to store the data dictionary and the location index record corresponding to each data record.
[0039] Fourthly, this application provides a data decompression apparatus, comprising:
[0040] The acquisition module is used to acquire compressed data, wherein the compressed data is obtained by compressing N data records by the data compression device in the third aspect, and the compressed data includes a data dictionary corresponding to the N data records and a position index record corresponding to each data record;
[0041] The decompression module is used to query the value of each field of each data record from the data dictionary according to the position index record corresponding to each data record, so as to obtain each data record.
[0042] Fifthly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the data compression method as described in the first aspect; or implement the steps of the data decompression method as described in the second aspect.
[0043] In a sixth aspect, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the data compression method as described in the first aspect; or implement the steps of the data decompression method as described in the second aspect.
[0044] In this embodiment, the data records are sorted according to the frequency of repetition of the same field value in the acquired data records, and a data dictionary is constructed to store the unique field values of each field. Then, by determining the position index of each field value in each data record within the data dictionary, each field value in each data record is converted into a corresponding position index, resulting in a position index record for each data record. By storing the data dictionary and the corresponding position index record for each data record, compression of multiple data records is achieved. Since the method of replacing field values in the data records with position indices in the data dictionary is adopted, regardless of the length of the field value, it can be replaced with a position index for storage, reducing compression complexity, improving compression effect, and increasing compression efficiency. Attached Figure Description
[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is a flowchart of a data compression method provided in an embodiment of this application;
[0047] Figure 2 This is a schematic diagram of the trie structure in an embodiment of this application;
[0048] Figure 3 This is a flowchart of a data decompression method provided in an embodiment of this application;
[0049] Figure 4 This is a schematic diagram of the structure of a data compression device provided in an embodiment of this application;
[0050] Figure 5 This is a schematic diagram of the structure of a data decompression device provided in an embodiment of this application;
[0051] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0052] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0053] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific sequence or order. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in sequences other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0054] The data compression method, apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.
[0055] Please refer to Figure 1 , Figure 1 This is a flowchart of a data compression method provided in an embodiment of this application. Figure 1 As shown, the data compression method includes the following steps:
[0056] Step 101: Obtain N data records. Each data record includes multiple field values, and each field value corresponds to a field.
[0057] It should be noted that the embodiments of this application can be applied to compress multiple data records, each with similar field values, and the character lengths of field values belonging to the same field within each data record are similar. Furthermore, the aforementioned field values can be integers, stored in character form, such as ID card numbers, mobile phone numbers, base station identification numbers (IDentity, ID), etc. Fields can represent the category to which field values belong, used to classify integers, and the field values corresponding to each category have similar lengths. Therefore, compared to existing methods for compressing common parts, the embodiments of this application can adapt to compression requirements and improve overall compression efficiency when compressing integer characters with a large number of common parts and a small number of characters.
[0058] Furthermore, the data to be compressed can include multiple data records, each containing multiple field values. Each field value can belong to a different field; for example, field value 14 belongs to the age field, and field value 110112 belongs to the ID number field. A single data record contains multiple field values belonging to different fields, as shown in Table 1.
[0059] Table 1
[0060] age ID number Base station ID Community ID Building ID 14 110112 35 27 13
[0061] Step 102: Construct a data dictionary based on the repetition frequency of field values corresponding to the same field in the N data records, wherein the data dictionary stores the unique field values for each field.
[0062] In this embodiment, the data dictionary records various field values. Field values for the same field can be arranged in a row within the data dictionary, and can be ordered according to the frequency of their recurrence within that field. Furthermore, no two field values in the same row are repeated; frequently occurring field values can be listed first or last. For example, constructing a data dictionary for each data record in Table 2 below yields the following: first row with field value 14, second row with field values 110114 and 110112, third row with field values 88, 35, and 44, fourth row with field values 27 and 45, and fifth row with field values 19, 13, and 18. This five-row data dictionary construction allows each field value in the data dictionary to be considered an index, and the data dictionary can include an index area. In this way, by arranging field values according to their recurrence frequency within the same field, the field values corresponding to each field in the data dictionary are determined. Since the data dictionary omits a large number of repetitive field values, it saves compression space. Furthermore, the field values in the data dictionary are arranged according to the frequency of repetition, which can quickly compress or decompress recurring data, improving compression efficiency and enhancing compression effect.
[0063] Table 2
[0064] age ID number Base station ID Community ID Building ID 14 110112 35 27 13 14 110114 88 45 19 14 110114 88 27 19 14 110114 44 45 18
[0065] Step 103: Determine the position index of each field value of each data record in the data dictionary, and convert each field value of each data record into the corresponding position index to obtain the position index record corresponding to each data record.
[0066] It's important to note that each data record contains multiple field values, each corresponding to a different field. The data dictionary also contains field values for different fields, with values for the same field arranged according to their frequency of occurrence within that field. Therefore, the position index of each field value in the data dictionary determines the position of the corresponding field value within the data record. This position index can be used to convert the field value into its corresponding position index. For example, if a data record contains the ID number 110112, and the corresponding row in the data dictionary contains the field values 110114 and 110112, then the position index for the ID number in this data record can be set to 1. If another data record contains the ID number 110114, then the position index for that ID number can be set to 0.
[0067] Furthermore, after determining the corresponding location index, each field value in each data record is converted into its corresponding location index, resulting in a location index record for each data record. For example, referring to the specific content of the data dictionary in the above embodiment, each data record in Table 2 can be replaced with each location index record in Table 3. Through the above steps, each data record can be converted into a location index record, enabling sufficient compression of the data records and reducing storage space requirements.
[0068] Table 3
[0069] age ID number Base station ID Community ID Building ID 0 0 0 1 0 0 1 1 0 1 0 0 0 0 0 0 0 2 1 2
[0070] Step 104: Store the data dictionary and the location index record corresponding to each data record.
[0071] Specifically, the data dictionary and the location index record corresponding to each data record can be stored as the compressed data of N data records. The data dictionary only includes unique field values, significantly reducing the data volume. Simultaneously, the location index record corresponding to each data record is also stored. By combining the location index record with the data dictionary, each data record can be reconstructed. Since the data volume of the location index record is reduced compared to the data record, the compression effect is improved, compression efficiency is increased, and the computational complexity of compression is reduced.
[0072] Optionally, after obtaining N data records and before storing the data dictionary and the position index record corresponding to each data record, the method further includes:
[0073] A baseline data record is determined based on the frequency of repetition of each field value in the corresponding field of each data record. The baseline data record is the data record with the highest sum of the repetition frequencies of each field value among the N data records.
[0074] After determining the position index of each field value of each data record in the data dictionary, and before storing the data dictionary and the position index record corresponding to each data record, the method further includes:
[0075] An index area is constructed based on the difference information between the position index record corresponding to other data records and the position index record corresponding to the benchmark data record. The index area stores the position index record corresponding to the benchmark data record and the difference index record of the other data records relative to the benchmark data record. The other data records are the data records other than the benchmark data record among the N data records.
[0076] The storage of the data dictionary and the location index record corresponding to each data record includes:
[0077] Store the data dictionary and the index area.
[0078] In one specific embodiment of this application, a baseline data record can be determined from multiple data records. This baseline data record can be the one with the highest sum of the repetition frequencies of all field values among the N data records. For example, the baseline data record is determined from the multiple data records in Table 4. The record ID in Table 4 may not exist in any of the data records. It can be determined that the character values corresponding to the age field are consistent. In the ID card number field, the character value 110114 appears most frequently. In the baseline ID field, the most frequently repeated value is 88. In the community ID field, the field values 27 and 45 appear with the same frequency. Finally, in the building ID field, the most frequently repeated value is 19. Since the second data record contains several relatively frequent field values such as 14, 110114, 88, 45, and 19, the second data record can be determined as the baseline data record. After determining the baseline data record, it can be used as the first data record in the data records (as shown in Table 5).
[0079] Furthermore, when dealing with a large number of data records, a trie approach can be used. Each field value is designated as a node in the trie, and the data record corresponding to a path with many common nodes can be identified as the baseline data record. In this way, the baseline data record can be one that is similar to all other data records among the N data records, serving as the basis for comparison with the other data records.
[0080] It is also worth mentioning that, in this application, multiple baseline data records can be set when the amount of data to be compressed is large. Specifically, this can be set according to the number of common nodes. If the preset number of common nodes is K, then if there are more than K common nodes in a path, that path can be designated as a baseline data record. By selecting multiple baseline data records when the data volume is large, computational complexity can be greatly reduced, storage space requirements can be reduced, and storage efficiency can be improved.
[0081] Table 4
[0082]
[0083]
[0084] Table 5
[0085] Record ID age ID number Base station ID Community ID Building ID 2 14 110114 88 45 19 1 14 110112 35 27 13 3 14 110114 88 27 19 4 14 110114 44 45 18
[0086] Furthermore, in this embodiment, each field value of each data record can be converted into a position index corresponding to the field value in the data dictionary. Both the base data record and other data records can be converted into position index records. The base data record can be retained, and a difference index record can be obtained between the position index record corresponding to the base data record and the position index records corresponding to other data records. An index area is constructed using this difference index record and the position index record corresponding to the base index record. In this case, the amount of data in the index area is relatively small, and each data record can be accurately located and restored. Regardless of the length of each field value in other data records, they can all be converted into smaller characters, reducing the compression space requirement and improving the compression effect.
[0087] For example, first determine the location index records corresponding to other data records, and the location index record corresponding to the baseline data record. Based on the specific content of the data records in Table 4, the data dictionary can be determined as follows: 14
[0089] 110114, 110112
[0090] 88,35,44
[0091] 27,45
[0092] 19, 13, 18.
[0093] The data dictionary can contain plain text. Each row in the data dictionary corresponds to a field in a data record, and each row includes unique field values.
[0094] Referring to the field values of each data record in Table 5, locate the position index of each field value in the data dictionary for the baseline data record. For example, if the baseline data records are 14, 110114, 88, 45, and 19, the age field corresponding to 14 is in the first row of the data dictionary, and 14 is in the first position, so the position index of 14 can be determined as 0. The ID number field corresponding to 110114 is in the second row of the data dictionary, and 110114 is also in the first position, so the position index of 110114 can be determined as 0. The base station ID field corresponding to 88 is in the third row of the data dictionary, and 88 is in the first position, so the position index of 88 can be determined as 0. The cell ID field corresponding to 45 is in the fourth row of the data dictionary, and 45 is in the second position, so the position index of 45 can be determined as 1. The base station ID field corresponding to 19 is in the third row of the data dictionary, and 19 is in the first position, so the position index of 19 can be determined as 0. Therefore, the position index record corresponding to the baseline data record can be determined to be 0, 0, 0, 1, 0.
[0095] Similarly, the method for determining the location index record corresponding to the baseline data record can be used to determine the location index record corresponding to other data records, which will not be repeated here. Finally, the converted location index records can be obtained, as shown in Table 6 below.
[0096] Table 6
[0097] Record ID age ID number Base station ID Community ID Building ID 2 0 0 0 1 0 1 0 1 1 0 1 3 0 0 0 0 0 4 0 0 2 1 2
[0098] Thus, after determining the location index records corresponding to other data records and the location index records corresponding to the baseline data record, the difference between each other data record and the baseline data record can be calculated to obtain the differences between the location indexes corresponding to each other data record and the location indexes corresponding to the baseline data record. For example, in Table 6, there are differences between the location index records corresponding to the baseline data record and the location index records corresponding to record ID 1 in the location indexes corresponding to the ID number, baseline ID, community ID, and building ID. Only these differences need to be recorded. These differences can be recorded as 1,2,4:1┃3:-1, indicating that between the location index records corresponding to the data record with record ID 1 and the location index records corresponding to the baseline data record, the fields in columns 1, 2, and 4 need to be incremented by 1 based on the location index records corresponding to the baseline data record, and the field in column 3 needs to be decremented by 1 based on the location index records corresponding to the baseline data record. Thus, the difference information between the location index records corresponding to each other data record and the location index records corresponding to the baseline data record in Table 6 can be obtained as follows:
[0099] 1,2,4:1|3:-1
[0100] 3:-1
[0101] 2,4:2.
[0102] Understandably, in this embodiment, a data dictionary and an index area can be stored as compressed data. The index area may include difference information and position index records corresponding to the baseline data records. The position index records of each other data record can be restored through the index area. Combined with the data dictionary, the compressed data can be decompressed. The compressed data can be decompressed according to actual needs, which is simple to calculate, highly flexible, and improves compression and decompression efficiency.
[0103] Optionally, the field values corresponding to the same field in the N data records that are not repeated are stored in the same row of the data dictionary, and the field values of the same field stored in each row of the data dictionary are sorted according to the frequency of repetition; the field values of different fields in the N data records are stored in different rows of the data dictionary.
[0104] Determining the position index of each field value in the data dictionary for each data record includes:
[0105] Based on the corresponding row data of each field value in the data dictionary, the column number of each field value in the corresponding row data is determined as the position index.
[0106] Optionally, the field values of the same field stored in each row of the data dictionary are sorted in descending order of repetition frequency.
[0107] In one embodiment of this application, the construction of the number dictionary is based on the principle of Huffman coding, where data with higher frequency of occurrence should have smaller indices. If each data record corresponds to M fields, the data dictionary can be configured to have M rows, each corresponding to one field. Furthermore, each row in the data dictionary stores the field values of the same field, which can be sorted according to the frequency of occurrence of each field value in the data record. Values with higher repetition frequency can be placed at the beginning of a row in the data dictionary, making them easier to retrieve, facilitating calculations, and reducing computational complexity.
[0108] Furthermore, the column number of each field value in the corresponding row of data can be determined as the position index, as in the above embodiment where the field values in Table 5 are converted into position indices in Table 6. See the above embodiment for details, which will not be repeated here.
[0109] Optionally, determining the baseline data record based on the repetition frequency of each field value in the corresponding field of each data record includes:
[0110] Using each field value of each data record as a node, construct a trie corresponding to the N data records, and merge duplicate field values of the same field into the same node during the construction of the trie. The trie includes N paths, and each path corresponds to one data record among the N data records.
[0111] Obtain the repetition frequency of each node in each path, and determine the sum of the repetition frequencies of each node in each path;
[0112] The data record corresponding to the path with the highest sum of repetition frequencies of all nodes among the N paths is determined as the baseline data record.
[0113] Please see Figure 2 , Figure 2 This is a schematic diagram of the trie structure in an embodiment of this application. The baseline data record can be determined by constructing a trie. Each field value represents a node in the trie representing a path; duplicate nodes in different paths can be merged into a single node. See Table 4 and... Figure 3 Specifically, among the four paths corresponding to the four data records in Table 4, the data record with ID 2 corresponding to the path with the most repeated nodes can be identified as the baseline data record. Using a trie, the frequency of repetition of each node in every other path can be clearly and intuitively displayed, allowing for quick identification of the baseline data record among multiple data records with relatively low computational complexity.
[0114] Optionally, constructing the index area based on the difference information between the location index record corresponding to other data records and the location index record corresponding to the reference data record includes:
[0115] Identify the target field in the location index record corresponding to the first data record that differs from the location index record corresponding to the reference data record, and determine the location index difference corresponding to the target field, wherein the first data record is any one of the other data records;
[0116] The sequence number of the target field and the position index difference are associated and written into the index area.
[0117] In another embodiment of this application, the sequence number and location index difference of the target field are associated and written into the index area. That is, the field that differs between each data record and the reference data record is its target field. There can be multiple target fields, which can be the same as the number of fields in the first data record. At this time, the value of each field in the first data record is different from the values of each field in the reference data record. As described in the above embodiment, there are differences between the location index record corresponding to the reference data record in Table 6 and the location index record corresponding to the data record with record ID 1 in terms of the location indexes corresponding to the ID card number, reference ID, community ID, and building ID. Only these differences can be recorded. The sequence numbers of the target fields such as ID card number, reference ID, community ID, and building ID are determined as 1, 2, 3, and 4, respectively, and the associated index differences are recorded as 1:1, 2:1, 4:1┃3:-1. This embodiment can locate the index differences existing in each data record, record only the target fields that differ, reduce storage space, reduce compression complexity, and improve compression effect.
[0118] Optionally, the step of associating the sequence number of the target field and the position index difference and writing it into the index area includes:
[0119] If multiple target fields in the first data record have the same position index difference, the multiple target fields are associated with one of the position index differences and written into the index area.
[0120] Specifically, when the position index differences corresponding to multiple target fields in the first data are consistent, only one position index difference and the correspondence between multiple target fields need to be recorded, thus saving compression space. For example, in Table 6, the position index record corresponding to the baseline data record and the position index record corresponding to the data record with record ID 1 have consistent target field sequence numbers of 1, 2, and 4. 1,2,4:1┃3:-1 can be written into the index area to save compression space and improve the compression effect.
[0121] Please see Figure 3 , Figure 3 A flowchart of a data decompression method provided in this application embodiment specifically includes the following steps:
[0122] Step 201: Obtain compressed data, wherein the compressed data is obtained by compressing N data records using the above data compression method embodiment, and the compressed data includes a data dictionary corresponding to the N data records and a position index record corresponding to each data record;
[0123] Step 202: Based on the location index record corresponding to each data record, query the value of each field of each data record from the data dictionary to obtain each data record.
[0124] Optionally, when the compressed data includes a data dictionary and an index area, the step of querying the data dictionary for each field value of each data record based on the position index record corresponding to each data record to obtain each data record includes:
[0125] Based on the position index record corresponding to the baseline data record stored in the index area, and the difference index record of other data records relative to the baseline data record, the position index record corresponding to the other data record is obtained, wherein the other data record is the data record other than the baseline data record among the N data records;
[0126] Based on the location index record corresponding to the baseline data record and the location index records corresponding to the other data records, the baseline data record is obtained by querying each field value of the baseline data record from the data dictionary, and the other data records are obtained by querying each field value of each data record in the other data records from the data dictionary.
[0127] In one specific embodiment of this application, decompressing the compressed data obtained using the aforementioned data compression method can be achieved by restoring the position index record corresponding to each data record by identifying the difference information and the corresponding position record of the baseline data record in the compressed data. Since the data dictionary contains multiple rows of field values, each row corresponding to a field in a data record, the field value corresponding to each position index in each position index record is found in the data dictionary, and each position index is restored to its field value. In this way, each data record can be obtained, which is then the decompressed data record, including the baseline data record and other data records, enabling fast and accurate data decompression.
[0128] In addition, some specific calculation operations for decompression can be the same as those for compression, and will not be repeated here. For details, please refer to the relevant descriptions in the foregoing embodiments.
[0129] This application also provides a data compression device, such as... Figure 4 As shown, the data compression device 300 specifically includes:
[0130] The data acquisition module 301 is used to acquire N data records, each data record including multiple field values, and each field value corresponds to a field;
[0131] The dictionary construction module 302 is used to construct a data dictionary based on the repetition frequency of field values corresponding to the same field in the N data records, wherein the data dictionary stores the unique field values for each field;
[0132] The location index module 303 is used to determine the location index of each field value of each data record in the data dictionary, and convert each field value of each data record into the corresponding location index to obtain the location index record corresponding to each data record;
[0133] The record storage module 304 is used to store the data dictionary and the location index record corresponding to each data record.
[0134] Optionally, the data acquisition module 301 further includes:
[0135] The benchmark determination submodule is used to determine the benchmark data record based on the repetition frequency of each field value in the corresponding field of each data record, wherein the benchmark data record is the data record with the highest sum of the repetition frequencies of each field value among the N data records;
[0136] The location index module 303 also includes:
[0137] The index area submodule is used to construct an index area based on the difference information between the position index record corresponding to other data records and the position index record corresponding to the reference data record. The index area stores the position index record corresponding to the reference data record and the difference index record of the other data records relative to the reference data record. The other data records are the data records other than the reference data record among the N data records.
[0138] Record storage module 304 includes:
[0139] The storage submodule is used to store the data dictionary and the index area.
[0140] Optionally, the field values corresponding to the same field in the N data records that are not repeated are stored in the same row of the data dictionary, and the field values of the same field stored in each row of the data dictionary are sorted according to the frequency of repetition; the field values of different fields in the N data records are stored in different rows of the data dictionary.
[0141] Location index module 303 includes:
[0142] The location index submodule is used to determine the column number of each field value in the corresponding row data in the data dictionary as the location index, based on the corresponding row data of each field value in each data record.
[0143] Optionally, the field values of the same field stored in each row of the data dictionary are sorted in descending order of repetition frequency.
[0144] Optionally, the benchmark determination submodule includes:
[0145] A trie unit is used to construct a trie corresponding to the N data records with each field value of each data record as a node, and to merge duplicate field values of the same field into the same node during the construction of the trie. The trie includes N paths, and each path corresponds to one data record among the N data records.
[0146] The node frequency unit is used to obtain the repetition frequency of each node in each path and determine the sum of the repetition frequencies of each node in each path.
[0147] The record determination unit is used to determine the data record corresponding to the path with the highest sum of repetition frequencies of all nodes in the N paths as the reference data record.
[0148] Optionally, the index area submodule includes:
[0149] The target field unit is used to determine the target field in which the position index of the position index record corresponding to the first data record differs from that of the position index record corresponding to the reference data record, and to determine the position index difference corresponding to the target field, wherein the first data record is any one of the other data records;
[0150] The writing unit is used to associate the sequence number of the target field with the position index difference and write it into the index area.
[0151] Optionally, the writing unit is used for:
[0152] If multiple target fields in the first data record have the same position index difference, the multiple target fields are associated with one of the position index differences and written into the index area.
[0153] The data compression device 300 provided in this application embodiment can achieve... Figure 1 The various processes implemented in the Chinese method embodiment achieve the same beneficial effects, and will not be described again here to avoid repetition.
[0154] Please see Figure 5 This application also provides a data decompression device 400, specifically including:
[0155] The acquisition module 401 is used to acquire compressed data, wherein the compressed data is obtained by compressing N data records by a data compression device, and the compressed data includes a data dictionary corresponding to the N data records and a position index record corresponding to each data record;
[0156] The decompression module 402 is used to query the value of each field of each data record from the data dictionary according to the position index record corresponding to each data record, so as to obtain each data record.
[0157] Optionally, if the compressed data includes a data dictionary and an index area, the decompression module 402 includes:
[0158] The first acquisition submodule is used to acquire the position index record corresponding to the other data record based on the position index record corresponding to the benchmark data record stored in the index area, and the difference index record of other data records relative to the benchmark data record, wherein the other data record is the data record other than the benchmark data record among the N data records;
[0159] The second acquisition submodule is used to query each field value of the benchmark data record from the data dictionary based on the location index record corresponding to the benchmark data record and the location index record corresponding to the other data records, and to obtain the benchmark data record by querying each field value of each data record in the other data records from the data dictionary.
[0160] The data decompression device 400 provided in this application embodiment can achieve... Figure 3 The various processes implemented in the Chinese method embodiment achieve the same beneficial effects, and will not be described again here to avoid repetition.
[0161] Please see Figure 6 , Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the electronic device includes: a processor 500, a memory 520, and a program or instructions stored in the memory 520 and executable on the processor 500. The processor 500 is used to read the program or instructions from the memory 520. The electronic device also includes a bus interface and a transceiver 510.
[0162] Transceiver 510 is used to receive and send data under the control of processor 500.
[0163] Among them, Figure 6In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 may be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 during operation.
[0164] The processor 500 is used to read programs or instructions from the memory 520 and execute the following steps:
[0165] Retrieve N data records, each data record including multiple field values, each field value corresponding to a field;
[0166] A data dictionary is constructed based on the frequency of repetition of the field values corresponding to the same field in the N data records, wherein the data dictionary stores the unique field values for each field;
[0167] Determine the position index of each field value of each data record in the data dictionary, and convert each field value of each data record into the corresponding position index to obtain the position index record corresponding to each data record;
[0168] Store the data dictionary and the location index record corresponding to each data record.
[0169] Optionally, the processor 500 is configured to read a program or instruction from the memory 520 and perform the following steps:
[0170] A baseline data record is determined based on the frequency of repetition of each field value in the corresponding field of each data record. The baseline data record is the data record with the highest sum of the repetition frequencies of each field value among the N data records.
[0171] An index area is constructed based on the difference information between the position index record corresponding to other data records and the position index record corresponding to the benchmark data record. The index area stores the position index record corresponding to the benchmark data record and the difference index record of the other data records relative to the benchmark data record. The other data records are the data records other than the benchmark data record among the N data records.
[0172] Store the data dictionary and the index area.
[0173] Optionally, the field values corresponding to the same field in the N data records that are not repeated are stored in the same row of the data dictionary, and the field values of the same field stored in each row of the data dictionary are sorted according to the frequency of repetition; the field values of different fields in the N data records are stored in different rows of the data dictionary.
[0174] Processor 500 is used to read programs or instructions from memory 520 and perform the following steps:
[0175] Based on the corresponding row data of each field value in the data dictionary, the column number of each field value in the corresponding row data is determined as the position index.
[0176] Optionally, the field values of the same field stored in each row of the data dictionary are sorted in descending order of repetition frequency.
[0177] Optionally, the processor 500 is configured to read a program or instruction from the memory 520 and perform the following steps:
[0178] Using each field value of each data record as a node, construct a trie corresponding to the N data records, and merge duplicate field values of the same field into the same node during the construction of the trie. The trie includes N paths, and each path corresponds to one data record among the N data records.
[0179] Obtain the repetition frequency of each node in each path, and determine the sum of the repetition frequencies of each node in each path;
[0180] The data record corresponding to the path with the highest sum of repetition frequencies of all nodes among the N paths is determined as the baseline data record.
[0181] Optionally, the processor 500 is configured to read a program or instruction from the memory 520 and perform the following steps:
[0182] Identify the target field in the location index record corresponding to the first data record that differs from the location index record corresponding to the reference data record, and determine the location index difference corresponding to the target field, wherein the first data record is any one of the other data records;
[0183] The sequence number of the target field and the position index difference are associated and written into the index area.
[0184] Optionally, the processor 500 is configured to read a program or instruction from the memory 520 and perform the following steps:
[0185] If multiple target fields in the first data record have the same position index difference, the multiple target fields are associated with one of the position index differences and written into the index area.
[0186] The electronic device provided in this application embodiment can perform the above-described functions. Figure 1 The various embodiments of the data compression method are similar in principle and technical effect, and will not be described again here.
[0187] This application also provides an electronic device. Since the principle by which the electronic device solves the problem is similar to the data compression method in this application, the implementation of this electronic device can refer to the implementation of the method, and repeated details will not be described again. The terminal in this application includes: a processor, configured to read a program from a memory and execute the following processes:
[0188] The processor is used to read programs from memory and execute the following procedures:
[0189] Obtain compressed data, wherein the compressed data is obtained by compressing N data records using the aforementioned electronic device, and the compressed data includes a data dictionary corresponding to the N data records and a position index record corresponding to each data record;
[0190] Based on the location index record corresponding to each data record, the value of each field of each data record is queried from the data dictionary to obtain each data record.
[0191] A transceiver is used to receive and send data under the control of a processor.
[0192] The bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors (represented by processors) and memories (represented by memory). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. A transceiver can be multiple components, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.
[0193] The processor is responsible for managing the bus architecture and general processing, while the memory stores the data used by the processor during operation.
[0194] Optionally, the processor may also read from memory and perform the following steps:
[0195] Based on the position index record corresponding to the baseline data record stored in the index area, and the difference index record of other data records relative to the baseline data record, the position index record corresponding to the other data record is obtained, wherein the other data record is the data record other than the baseline data record among the N data records;
[0196] Based on the location index record corresponding to the baseline data record and the location index records corresponding to the other data records, the baseline data record is obtained by querying each field value of the baseline data record from the data dictionary, and the other data records are obtained by querying each field value of each data record in the other data records from the data dictionary.
[0197] The electronic device provided in this application embodiment can perform the above-described functions. Figure 3 The various embodiments of the data decompression method are similar in principle and technical effect, and will not be described again here.
[0198] This application embodiment also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the above-described functionality. Figure 1 Various processes or implementations of the data compression method in China Figure 3 The various processes of the data decompression method can achieve the same technical effect, and to avoid repetition, they will not be described in detail here.
[0199] The processor is the processor in the electronic device described in the above embodiments. A readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0200] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0201] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0202] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.
[0203] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data compression method, characterized in that, include: Retrieve N data records, each data record including multiple field values, each field value corresponding to a field; A data dictionary is constructed based on the frequency of repetition of the field values corresponding to the same field in the N data records, wherein the data dictionary stores the unique field values for each field; Determine the position index of each field value of each data record in the data dictionary, and convert each field value of each data record into the corresponding position index to obtain the position index record corresponding to each data record; Store the data dictionary and the location index record corresponding to each data record; After obtaining N data records and before storing the data dictionary and the position index record corresponding to each data record, the method further includes: A baseline data record is determined based on the frequency of repetition of each field value in the corresponding field of each data record. The baseline data record is the data record with the highest sum of the repetition frequencies of each field value among the N data records. After determining the position index of each field value of each data record in the data dictionary, and before storing the data dictionary and the position index record corresponding to each data record, the method further includes: An index area is constructed based on the difference information between the position index record corresponding to other data records and the position index record corresponding to the benchmark data record. The index area stores the position index record corresponding to the benchmark data record and the difference index record of the other data records relative to the benchmark data record. The other data records are the data records other than the benchmark data record among the N data records. The storage of the data dictionary and the location index record corresponding to each data record includes: Store the data dictionary and the index area.
2. The method according to claim 1, characterized in that, The field values corresponding to the same field in the N data records and which are not repeated are stored in the same row of the data dictionary, and the field values of the same field stored in each row of the data dictionary are sorted according to the frequency of repetition. The field values of different fields in the N data records are stored in different rows of the data dictionary; Determining the position index of each field value in the data dictionary for each data record includes: Based on the corresponding row data of each field value in the data dictionary, the column number of each field value in the corresponding row data is determined as the position index.
3. The method according to claim 2, characterized in that, The field values of the same field stored in each row of the data dictionary are sorted in descending order of repetition frequency.
4. The method according to claim 1, characterized in that, The step of determining the baseline data record based on the repetition frequency of each field value in the corresponding field of each data record includes: Using each field value of each data record as a node, construct a trie corresponding to the N data records, and merge duplicate field values of the same field into the same node during the construction of the trie. The trie includes N paths, and each path corresponds to one data record among the N data records. Obtain the repetition frequency of each node in each path, and determine the sum of the repetition frequencies of each node in each path; The data record corresponding to the path with the highest sum of repetition frequencies of all nodes among the N paths is determined as the baseline data record.
5. The method according to claim 1, characterized in that, The step of constructing an index area based on the difference information between the position index record corresponding to other data records and the position index record corresponding to the reference data record includes: Identify the target field in the location index record corresponding to the first data record that differs from the location index record corresponding to the reference data record, and determine the location index difference corresponding to the target field, wherein the first data record is any one of the other data records; The sequence number of the target field and the position index difference are associated and written into the index area.
6. The method according to claim 5, characterized in that, The step of associating the sequence number of the target field with the position index difference and writing it into the index area includes: If multiple target fields in the first data record have the same position index difference, the multiple target fields are associated with one of the position index differences and written into the index area.
7. A data decompression method, characterized in that, include: Obtain compressed data, wherein the compressed data is obtained by compressing N data records using the data compression method according to any one of claims 1 to 6, and the compressed data includes a data dictionary corresponding to the N data records and a position index record corresponding to each data record; Based on the location index record corresponding to each data record, query the value of each field of each data record from the data dictionary to obtain each data record; When the compressed data includes a data dictionary and an index area, the step of retrieving the value of each field of each data record from the data dictionary based on the position index record corresponding to each data record to obtain each data record includes: Based on the position index record corresponding to the baseline data record stored in the index area, and the difference index record of other data records relative to the baseline data record, the position index record corresponding to the other data record is obtained, wherein the other data record is the data record other than the baseline data record among the N data records; Based on the location index record corresponding to the baseline data record and the location index records corresponding to the other data records, the baseline data record is obtained by querying each field value of the baseline data record from the data dictionary, and the other data records are obtained by querying each field value of each data record in the other data records from the data dictionary.
8. A data compression device, characterized in that, include: The data acquisition module is used to acquire N data records. Each data record includes multiple field values, and each field value corresponds to a field. A dictionary construction module is used to construct a data dictionary based on the repetition frequency of field values corresponding to the same field in the N data records, wherein the data dictionary stores the unique field values for each field; The location index module is used to determine the location index of each field value of each data record in the data dictionary, and convert each field value of each data record into the corresponding location index to obtain the location index record corresponding to each data record; A record storage module is used to store the data dictionary and the location index record corresponding to each data record; The data acquisition module further includes: The benchmark determination submodule is used to determine the benchmark data record based on the repetition frequency of each field value in the corresponding field of each data record, wherein the benchmark data record is the data record with the highest sum of the repetition frequencies of each field value among the N data records; The location indexing module further includes: The index area submodule is used to construct an index area based on the difference information between the position index record corresponding to other data records and the position index record corresponding to the reference data record. The index area stores the position index record corresponding to the reference data record and the difference index record of the other data records relative to the reference data record. The other data records are the data records other than the reference data record among the N data records. The record storage module includes: The storage submodule is used to store the data dictionary and the index area.
9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data compression method as described in any one of claims 1 to 6 or the data decompression method as described in claim 7.
10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data compression method as described in any one of claims 1 to 6 or the data decompression method as described in claim 7.