Data table storage method and device, computer storage medium and electronic equipment

Abstract

The invention relates to the technical field of computers, and provides a data table storage method, a data table storage device, a computer storage medium and electronic device.The data table storage method comprises the steps that information of M data tables needing to be compressed is read from a preset compression configuration file; m is an integer greater than 1; obtaining the M data tables according to the information of the M data tables needing to be compressed; compressing and merging the M data tables according to compression and merging demand information contained in the compression configuration file to obtain a compression and merging result; the compression and merging demand information comprises a service main key corresponding to each data table in the M data tables and at least one to-be-compressed field corresponding to each data table; and recompressing the compression and combination result, and storing the obtained recompression result to a specified storage system. According to the method, the data storage performance can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and in particular, to a data table storage method, a data table storage device, a computer storage medium and an electronic device. BACKGROUND

[0002] In the Internet industry, especially in the logistics industry, real-time data is the strategic basis for the development of each enterprise for the top management. For real-time processing and storage of massive data, how to effectively improve the performance of data storage and provide real-time and fast data for the top management is particularly important.

[0003] In the related art, massive data is directly stored in the storage system in real time without processing, resulting in poor storage performance.

[0004] Therefore, there is an urgent need in the art to develop a new data table storage method and device.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure. SUMMARY

[0006] The purpose of the present disclosure is to provide a data table storage method, a data table storage device, a computer storage medium and an electronic device, thereby at least partially overcoming the technical problem of poor storage performance caused by the limitations of the related art.

[0007] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a data table storage method is provided, comprising:

[0009] reading information of M data tables to be compressed from a pre-set compression configuration file; M is an integer greater than 1;

[0010] obtaining the M data tables according to the information of the M data tables to be compressed;

[0011] performing compression and merging on the M data tables according to compression and merging requirement information contained in the compression configuration file to obtain a compression and merging result; the compression and merging requirement information contains a business primary key corresponding to each data table in the M data tables and at least one field to be compressed corresponding to each data table;

[0012] re-compressing the compression and merging result, and storing the obtained re-compression result in a designated storage system.

[0013] In the example embodiment of the present disclosure, the compressing and merging of the M data tables according to the compression and merging requirement information contained in the compression configuration file to obtain a compression and merging result comprises:

[0014] The field values under each to-be-compressed field of each data table are subjected to field value deduplication and merging to obtain a target data table;

[0015] At least one data table set with the same business primary key is selected from the M target data tables, and each data table set contains at least two target data tables;

[0016] The at least two target data tables are subjected to data deduplication and merging to merge the at least two target data tables into an integrated data table;

[0017] At least one integrated data table corresponding to the at least one data table set is determined as the compression and merging result.

[0018] In the example embodiment of the present disclosure, the data deduplication and merging of the at least two target data tables to merge the at least two target data tables into an integrated data table comprises:

[0019] The business primary key corresponding to each data table set is determined as the business primary key corresponding to the integrated data table;

[0020] The at least two to-be-compressed fields corresponding to the at least two target data tables are subjected to field deduplication and merging, and the field values corresponding to the at least two to-be-compressed fields are subjected to field value deduplication and merging to merge the at least two target data tables into an integrated data table.

[0021] In the example embodiment of the present disclosure, the data deduplication and merging of the at least two target data tables to merge the at least two target data tables into an integrated data table comprises:

[0022] A joint business primary key configured according to a business requirement is determined as the business primary key corresponding to the integrated data table, and the joint business primary key is selected from at least two to-be-compressed fields corresponding to the at least two target data tables;

[0023] The at least two to-be-compressed fields corresponding to the at least two target data tables are subjected to field deduplication and merging, and the field values corresponding to the at least two to-be-compressed fields are subjected to field value deduplication and merging to merge the at least two target data tables into an integrated data table.

[0024] In the example embodiment of the present disclosure, the re-compression of the compression and merging result comprises:

[0025] According to a preset data filtering rule, invalid data in the compression and merging result is subjected to data filtering processing.

[0026] In the example embodiment of the present disclosure, the specified storage system comprises a plurality of data storage queues, and different data storage queues correspond to different queue names.

[0027] The storing of the obtained re-compression result to the specified storage system comprises:

[0028] According to a preset parsing rule, the field value under each field in each of the integrated data tables is parsed into a preset mapping value; and different preset mapping values are used to indicate the queue names of different storage queues in the specified storage system.

[0029] The field and the field value are stored in the data storage queue corresponding to the queue name indicated by the preset mapping value.

[0030] In the example embodiment of the present disclosure, the preset parsing rule is used to parse the field values contained in the same field into the same preset mapping value.

[0031] According to a second aspect of the present disclosure, a data table storage device is provided, comprising:

[0032] An information reading module is configured to read information of M data tables to be compressed from a preset compression configuration file; M is an integer greater than 1.

[0033] A data table obtaining module is configured to obtain the M data tables according to the information of the M data tables to be compressed.

[0034] A compression and merging module is configured to perform compression and merging on the M data tables according to compression and merging requirement information contained in the compression configuration file, to obtain a compression and merging result; the compression and merging requirement information comprises a business primary key corresponding to each of the M data tables and at least one field to be compressed corresponding to each of the M data tables.

[0035] A re-compression module is configured to perform re-compression on the compression and merging result, and store the obtained re-compression result to a specified storage system.

[0036] According to a third aspect of the present disclosure, a computer storage medium is provided, which stores a computer program; when the computer program is executed by a processor, the data table storage method of the first aspect is implemented.

[0037] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data table storage method of the first aspect described above via execution of the executable instructions.

[0038] From the above technical solutions, the data table storage method, the data table storage device, the computer storage medium and the electronic device in the example embodiments of the present disclosure at least have the following advantages and positive effects:

[0039] In the technical solutions provided by some embodiments of the present disclosure, on the one hand, by reading the information of M data tables to be compressed from the pre-set compression configuration file, obtaining the M data tables according to the information of the M data tables to be compressed, and then performing compression and merging on the M data tables according to the compression and merging requirement information (the compression and merging requirement information includes the business primary key corresponding to each data table in the M data tables and at least one field to be compressed corresponding to each data table) contained in the compression configuration file, the compression and merging result is obtained, which can merge the data tables with correlations in the data source system, thereby avoiding the problem that in the related art, if the associated data of a data table needs to be obtained in the data processing stage, the interface needs to be called across business lines to obtain the data, so that in the peak period of massive data, the interface may be frequently called, resulting in data backlog and low data processing efficiency, and a series of problems of calling the interface to obtain data across business lines are avoided, which provides a guarantee for the improvement of subsequent data processing efficiency. On the other hand, by re-compressing the compression and merging result and storing the obtained re-compression result to the specified storage system, the storage IO (Input / Output) can be reduced by re-compression, the storage efficiency is improved, and the data storage performance is optimized.

[0040] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0042] Figure 1 A flowchart of a data table storage method in an embodiment of the present disclosure is shown;

[0043] Figure 2A flowchart showing the process of compressing and merging M data tables according to the compression and merging requirement information contained in the compression profile to obtain a compression and merging result in the embodiment of the present disclosure is shown.

[0044] Figure 3 A flowchart showing the process of storing the obtained recompression result to a designated storage system in the embodiment of the present disclosure is shown.

[0045] Figure 4 A general architecture diagram of the data table processing method in the embodiment of the present disclosure is shown.

[0046] Figure 5 A structural diagram of a one-layer compression device in a two-layer compression device in the embodiment of the present disclosure is shown.

[0047] Figure 6 A structural diagram of a two-layer compression device in a two-layer compression device in the embodiment of the present disclosure is shown.

[0048] Figure 7 A structural diagram of a data table storage device in the exemplary embodiment of the present disclosure is shown.

[0049] Figure 8 A structural diagram of an electronic device in the exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0051] In this specification, the use of the phrases "one", "a", "the", and "said” is used to indicate that there is one or more of the elements / components / etc.; the use of the phrases “include” and “have” is used to indicate an open-ended inclusion and means that additional elements / components / etc. can be present in addition to the listed elements / components / etc.; the use of the phrases “first” and “second” etc. is used only as labels and is not a limitation on the number of objects.

[0052] In addition, the accompanying drawings are only schematic and are non-limiting illustrative of the disclosure. Identical reference signs denote identical or similar parts throughout the figures. Some of the blocks in the drawings are functional entities that may not necessarily have a corresponding physical or logical implementation.

[0053] At present, when corresponding business processing and processing are performed on data consumption of each production link, for business data across business lines, while processing multiple data table sources, the data of non-current data table needs to be obtained by calling an interface, and a large amount of data needs to be processed and randomly stored into a data source wide table.

[0054] Among them, the wide table is a table with more fields, containing more dimension levels, resulting in more redundancy. It usually refers to a database table in which business theme related indicators, dimensions and attributes are associated together. Since different contents are stored in the same table, the wide table does not conform to the model design specification of the three normal forms, and the main disadvantage is the large amount of data redundancy, and the corresponding advantage is the improvement and convenience of query performance. The design of such wide table is widely used in data preparation before data mining model training, and by placing related fields in the same table, the efficiency problem in the iterative calculation process of data mining model training can be greatly improved.

[0055] However, the above scheme has the following disadvantages:

[0056] On the one hand, when all data is stored, for multiple data tables, the data tables across business lines need to communicate with each other, frequently call interfaces, and the efficiency is very low, which is easy to cause data backlog; on the other hand, when mass data is stored, no corresponding merging and compression processing is performed, which will cause high disk IO consumption and poor storage performance.

[0057] The present disclosure aims to perform different rule merging and compression from two links of the whole data flow processing process, to improve the data processing and storage efficiency. Specifically, on the one hand, in the link of receiving data by the data source system, the message layer compression device is used to recombine the multi-topic data according to the compression and merging demand information, to perform first layer compression and merging, and on the other hand, in the link of batch storing data, the processing and storing layer compression device is used to classify and map the data after the first layer compression and merging according to fields, to perform merging and compression according to the mapping result, to complete second layer compression.

[0058] In the embodiments of the present disclosure, a data table storage method is first provided, which at least partly overcomes the defect of poor storage performance in the related art.

[0059] Figure 1A flowchart of a data table storage method in the embodiment of the present disclosure is shown. The execution subject of the data table storage method can be a server that stores data tables.

[0060] Reference Figure 1 The data table storage method according to one embodiment of the present disclosure includes the following steps:

[0061] In step S110, information of M data tables that need to be compressed is read from a pre-set compression configuration file; M is an integer greater than 1;

[0062] In step S120, the M data tables are obtained according to the information of the M data tables that need to be compressed;

[0063] In step S130, the M data tables are compressed and merged according to the compression and merging requirement information contained in the compression configuration file, and a compression and merging result is obtained; the compression and merging requirement information contains a business primary key corresponding to each data table in the M data tables and at least one field to be compressed corresponding to each data table;

[0064] In step S140, the compression and merging result is re-compressed, and the obtained re-compression result is stored in a designated storage system.

[0065] In Figure 1 The technical solution provided in the embodiment shown above has the following advantages. On the one hand, by reading the information of M data tables that need to be compressed from a pre-set compression configuration file, obtaining the M data tables according to the information of the M data tables that need to be compressed, and then compressing and merging the M data tables according to the compression and merging requirement information (the compression and merging requirement information contains a business primary key corresponding to each data table in the M data tables and at least one field to be compressed corresponding to each data table) contained in the compression configuration file, a compression and merging result is obtained. This can merge data tables with correlations in a data source system, thereby avoiding the problem in related technologies that, in the data processing stage, if the correlation data of a data table needs to be obtained, an interface may need to be called across business lines to obtain the data, which may result in frequent calling of the interface during a peak period of massive data, causing data backlog and low data processing efficiency. This avoids a series of problems of calling an interface to obtain data across business lines and provides a guarantee for the improvement of subsequent data processing efficiency. On the other hand, by re-compressing the compression and merging result and storing the obtained re-compression result in a designated storage system, the storage IO can be reduced by re-compression, the storage efficiency can be improved, and the data storage performance can be optimized.

[0066] The specific implementation process of each step in Figure 1 is described in detail below:

[0067] Before step S110, it needs to be explained that the disclosure can be pre-set a compression configuration file, which can contain the following factors: information of M (M is an integer greater than 1) data tables that need to be compressed (for example: Topic name of the M data tables, Topic is similar to folder in file system) and compression and merging requirement information for the M data tables, which is used to indicate how to compress and merge the M data tables according to certain rules or ways.

[0068] Among them, the compression and merging requirement information can contain the business primary key corresponding to each of the M data tables and at least one to-be-compressed field corresponding to each of the M data tables, and can also contain information such as fields contained in each of the M data tables, which can be set by the actual demand, and the disclosure does not make special limitation thereto.

[0069] Among them, the business primary key is a unique identifier that can determine a data record, for example, a data record includes fields such as ID number, name, age, etc., and the ID number is the only one that can determine the person, and the others can be repeated, so the ID number is the business primary key. The role of the business primary key is to ensure the uniqueness and integrity of the data, and at the same time, through the business primary key search table, the retrieval speed can be improved.

[0070] For example, taking a data table related to logistics as an example, the above-mentioned business primary key can be a waybill number. The fields contained in each data table can be name, order time, sender address, receiver address, sender name, receiver name, etc. The to-be-compressed field can be any one or more of the above-mentioned multiple fields, which can be set by the actual situation, and the disclosure does not make special limitation thereto.

[0071] After the above-mentioned compression configuration file is configured, the following steps S110-S130 can be executed when the data source system receives data, and the following step S140 can be executed when entering the data storage stage.

[0072] In step S110, the information of M data tables that need to be compressed is read from the pre-set compression configuration file.

[0073] In this step, after the data source system receives the real-time data table Topic of each business link, the information of the M data tables that need to be compressed can be read from the pre-set compression configuration file.

[0074] In step S120, the M data tables are obtained according to the information of the M data tables that need to be compressed.

[0075] In this step, the data source system can obtain the M data tables from the real-time data table Topic according to the information of the M data tables that need to be compressed.

[0076] In step S130, the M data tables are compressed and merged according to the compression and merging requirement information contained in the compression configuration file, and a compression and merging result is obtained.

[0077] In this step, after obtaining the M data tables that need to be compressed, the data source system can compress and merge the M data tables according to the compression and merging requirement information contained in the compression configuration file, and obtain a compression and merging result.

[0078] Reference Figure 2 , Figure 2 A flowchart for compressing and merging the M data tables according to the compression and merging requirement information contained in the compression configuration file to obtain a compression and merging result is shown in the embodiments of the present disclosure, which includes steps S201-S204:

[0079] In step S201, the field values under each field of each data table that needs to be compressed are de-duplicated and merged to obtain a target data table.

[0080] In this step, for each single table in the M data tables, since N (N is an integer greater than 1) binlog data records are stored in each single table, and the N data records can contain different fields, each field corresponds to different field values. Therefore, the present disclosure can merge the field values of the data table that needs to be compressed according to the business primary key.

[0081] For example, assuming that the primary key of a data table is KEY1, and there are a total of 2 data records in the data table, the field values of the first data record for the field that needs to be compressed are A1, A2, A3, and A4, and the field values of the second data record for the field that needs to be compressed are A2, A4, A5, and A6. After de-duplication and merging, the target data table contains data records A1, A2, A3, A4, A5, and A6, and the primary key of the data record is still KEY1.

[0082] Similarly, the step S201 can be performed on the M data tables to merge multiple data records in each single table into one data record. By merging multiple data records into one data record, the number of data write operations can be reduced when storing data, thereby reducing the performance impact on the specified storage system.

[0083] After performing the step S201 on the M data tables, step S202 can be entered.

[0084] In step S202, at least one data table set with the same business primary key is selected from the M target data tables, and each data table set contains at least two target data tables.

[0085] In this step, it can be determined whether there is at least one data table set with the same business primary key in the M target data tables. Taking M as 10 as an example, assuming that the business primary keys of the 10 data tables are all the same, the at least one data table set contains 10 target data tables; assuming that the business primary keys of table 1-table 5 (assuming corresponding business primary key KEY1) and the business primary keys of table 6-table 10 (assuming corresponding business primary key KEY2) are the same in the 10 target data tables, two data table sets can be obtained, the first data table set contains table 1-table 5, and the second data table set contains table 6-table 10.

[0086] In step S203, data deduplication and merging are performed on the at least two target data tables to merge the at least two target data tables into one integrated data table.

[0087] In this step, after obtaining the at least one data table set, data deduplication and merging can be performed on the at least two target data tables in each data table set to merge the at least two target data tables into one integrated data table.

[0088] In an optional implementation, the business primary key corresponding to each data table set can be determined as the business primary key corresponding to the integrated data table; field deduplication and merging are performed on at least two to-be-compressed fields corresponding to the at least two target data tables, and field value deduplication and merging are performed on field values corresponding to the at least two to-be-compressed fields to merge the at least two target data tables into one integrated data table.

[0089] For example, assuming that the business primary keys corresponding to target data table 1 and target data table 2 are both KEY1, and the to-be-compressed field corresponding to target data table 1 is field P, and the to-be-compressed field corresponding to target data table 2 is field Q, the business primary key of the integrated data table can be KEY1, and the integrated data table contains field P and field Q, and the integrated data table also contains all field values under field P and all field values after deduplication under field Q.

[0090] By merging target data tables with the same business primary keys, the present disclosure can directly merge data with correlations into one data table in a data source system, thereby avoiding the problem in the related art that, in the data processing stage, if correlation data of a data table is needed, an interface may need to be called across business lines to obtain the data, which may result in frequent interface calling in a mass data peak period, causing data backlog and low data processing efficiency, and eliminating a series of problems of calling an interface to obtain data across business lines, thereby providing a guarantee for the improvement of subsequent data processing efficiency.

[0091] In another optional implementation, when merging data of multiple tables, the present disclosure also supports configuring a joint business primary key (the joint business primary key is selected from at least two to-be-compressed fields corresponding to at least two target data tables) according to business requirements, so that the joint business primary key configured according to business requirements can be determined as the business primary key of the integrated data table, the field de-duplication and merging of the at least two to-be-compressed fields corresponding to the at least two target data tables are performed, and the field value de-duplication and merging of the field values corresponding to the at least two to-be-compressed fields are performed, so as to merge the at least two target data tables into one integrated data table.

[0092] For example, it is still assumed that the business primary keys of the target data table 1 and the target data table 2 are both KEY1, and the to-be-compressed field of the target data table 1 is field P, and the to-be-compressed field of the target data table 2 is field Q, then the joint business primary key (the joint business primary key is a plurality of fields used as the primary key of a table) can be configured as field P+field Q, so that the business primary key of the integrated data table can be field P+field Q, and the integrated data table contains field P and field Q, and correspondingly, the integrated data table also contains all field values under field P and all field values after de-duplication of field Q.

[0093] By configuring the joint business primary key according to business requirements, the target data table 1 and the target data table 2 can be stored as multiple integrated data tables with different joint business primary keys, so as to facilitate subsequent quick calling of different integrated data tables according to business requirements or business purposes.

[0094] In step S204, at least one integrated data table corresponding to at least one data table set is determined as a compression merging result.

[0095] In this step, referring to the related explanations of step S203, at least one integrated data table corresponding to at least one data table set can be determined as a compression merging result.

[0096] After obtaining the compression merging result, then referring to Figure 1In step S140, the compressed and merged result is re-compressed, and the obtained re-compressed result is stored in a designated storage system.

[0097] In this step, invalid data in the compressed and merged result is filtered according to a pre-set data filtering rule.

[0098] In this step, the re-compression process can be a filtering process of data. The pre-set data filtering rule can be a rule associated with a data type. For example, an indication label of a data type corresponding to each data record can be obtained (for example, a label of 1 represents pure distribution data, and a label of 2 represents warehouse distribution data), so that invalid data in the compressed and merged result can be filtered according to the data filtering rule based on the indication label. For example, when the data to be stored is warehouse distribution data, the data filtering rule can be that warehouse distribution data is valid data, and data of other types is invalid data. Similarly, the data filtering rule can be set according to actual conditions, and the present disclosure does not make special limitations on this.

[0099] By filtering the compressed and merged result, the amount of subsequent data processing can be reduced.

[0100] After obtaining the re-compressed data through the above data filtering process, the re-compressed data can be stored in a designated storage system. Figure 3 , Figure 3 A flowchart for storing the obtained re-compressed result in a designated storage system in an embodiment of the present disclosure is shown, including steps S301-S302.

[0101] In step S301, the field value under each field in each integrated data table is parsed into a pre-set mapping value according to a pre-set parsing rule.

[0102] In this step, the pre-set parsing rule is used to parse the field values contained in the same field into the same pre-set mapping value. For example, all field values under the “waybill number” field are parsed into a, and all field values under the “sender” field are parsed into b. The pre-set parsing rule can be set according to actual conditions, and the present disclosure does not make special limitations on this.

[0103] For example, a pre-set string parsing tool can be called to parse the field value under each field in each integrated data table into a pre-set mapping value based on the pre-set parsing rule.

[0104] It should be noted that the designated storage system may be an ES storage system, which may contain multiple data storage queues, each with a different queue name. A data storage queue is a message queue, which is an asynchronous communication method for message subscription, production, and consumption, such as ActiveMQ, Kafka, and ZeroMQ. This configuration is customizable based on actual circumstances and is not specifically limited in this disclosure.

[0105] Thus, the above-mentioned different preset mapping values ​​are used to indicate the queue names of different storage queues in the above-mentioned specified storage system. For example, when the preset mapping value is a, the indicated queue name may be X1, and when the above-mentioned preset mapping value is b, the indicated queue name may be X2. The specific indication relationship can be set according to actual conditions, and the present disclosure does not make any special limitations on this.

[0106] In step S302, the fields and the field values ​​are stored correspondingly in the data storage queue corresponding to the queue name indicated by the preset mapping value.

[0107] In this step, the above-mentioned fields and field values ​​can be stored correspondingly in the data storage queue corresponding to the queue name indicated by the preset mapping value. Therefore, given that the preset mapping values ​​corresponding to all field values ​​under the same field are the same, the present disclosure can store all field values ​​under the same field in the same data storage queue, and classify the field values ​​corresponding to different fields into different data storage queues to realize the classified storage of data.

[0108] refer to Figure 4 , Figure 4 The overall architecture diagram of the data table processing method in the embodiment of the present disclosure is shown as follows: Figure 4 As shown, the data table processing method in the present disclosure can be implemented by a double-layer compression device. When the double-layer compression device receives the data table to be compressed, it can be compressed by a first-layer compression device and a second-layer compression device, and then the processed results are stored in the ES data source.

[0109] refer to Figure 5 , Figure 5 A schematic diagram showing the structure of a layer of the double-layer compression device in the embodiment of the present disclosure is shown. Figure 5 As shown, the data source layer can obtain the data table to be compressed, and the compression processing executor of the compression processing layer can compress and merge the data table based on the compression configuration file, and store the compression and merging results in the compression result layer.

[0110] refer to Figure 6 , Figure 6A structure diagram of a two-layer compression device in the double-layer compression device is shown in the embodiments of the present disclosure, as shown in Figure 6 The data filter of the data filtering layer can filter invalid data from the compression and merging result, the mapping device of the mapping storage layer can parse the field values under each field in the filtered compression and merging result into mapping values, and the compression processor of the compression processing layer can perform classified compression and merging on the fields and field values according to the mapping values and store the final processing result in the compression result storage queue of the compression result layer. Based on the above technical solutions, the present disclosure has at least the following technical effects:

[0111] First, by performing compression and merging on the data tables in the data source system, interface communication between multiple tables is avoided, the utilization space of effective data is improved, and the processing performance is improved.

[0112] Second, by re-compression in the data processing stage, the number of data storage times is reduced, the storage IO is greatly reduced, and the storage performance is efficiently improved.

[0113] Third, in the process of data compression and merging, joint primary keys and other information can be flexibly configured according to actual business requirements, so that the present disclosure can adapt to compression requirements in different business scenarios.

[0114] The present disclosure also provides a data table storage device, Figure 7 A structure diagram of a data table storage device in an exemplary embodiment of the present disclosure is shown, as shown in Figure 7 The data table storage device 700 can include an information reading module 710, a data table acquisition module 720, a compression and merging module 730, and a re-compression module 740. Wherein:

[0115] The information reading module 710 is configured to read information of M data tables to be compressed from a pre-set compression configuration file; M is an integer greater than 1;

[0116] The data table acquisition module 720 is configured to acquire the M data tables according to the information of the M data tables to be compressed;

[0117] The compression and merging module 730 is configured to perform compression and merging on the M data tables according to compression and merging requirement information contained in the compression configuration file, and obtain a compression and merging result; the compression and merging requirement information includes a business primary key corresponding to each data table in the M data tables and at least one field to be compressed corresponding to each data table;

[0118] The re-compression module 740 is configured to re-compress the compression and merging result and store the obtained re-compression result to a designated storage system.

[0119] In the example embodiment of the present disclosure, the compression and merging module 730 performs compression and merging on the M data tables according to the compression and merging requirement information contained in the compression profile, and obtains a compression and merging result, including:

[0120] The field values under each to-be-compressed field of each data table are subjected to field value deduplication and merging, and a target data table is obtained;

[0121] At least one data table set with the same business primary key is selected from the M target data tables, and each data table set contains at least two target data tables;

[0122] The at least two target data tables are subjected to data deduplication and merging to merge the at least two target data tables into an integrated data table;

[0123] At least one integrated data table corresponding to the at least one data table set is determined as the compression and merging result.

[0124] In the example embodiment of the present disclosure, the compression and merging module 730 performs data deduplication and merging on the at least two target data tables to merge the at least two target data tables into an integrated data table, including:

[0125] The business primary key corresponding to each data table set is determined as the business primary key corresponding to the integrated data table;

[0126] At least two to-be-compressed fields corresponding to the at least two target data tables are subjected to field deduplication and merging, and field values corresponding to the at least two to-be-compressed fields are subjected to field value deduplication and merging, so as to merge the at least two target data tables into an integrated data table.

[0127] In the example embodiment of the present disclosure, the compression and merging module 730 performs data deduplication and merging on the at least two target data tables to merge the at least two target data tables into an integrated data table, including:

[0128] A joint business primary key configured by the business requirement is determined as the business primary key corresponding to the integrated data table, and the joint business primary key is selected from at least two to-be-compressed fields corresponding to the at least two target data tables;

[0129] At least two to-be-compressed fields corresponding to the at least two target data tables are subjected to field deduplication and merging, and field values corresponding to the at least two to-be-compressed fields are subjected to field value deduplication and merging, so as to merge the at least two target data tables into an integrated data table.

[0130] In an example embodiment of the present disclosure, the re-compression module 740 re-compresses the compressed merge result, including:

[0131] According to a preset data filtering rule, the invalid data in the compressed merge result is subjected to data filtering processing.

[0132] In an example embodiment of the present disclosure, the specified storage system comprises a plurality of data storage queues, and different data storage queues correspond to different queue names.

[0133] The re-compression module 740 stores the obtained re-compression result to the specified storage system, including:

[0134] According to a preset parsing rule, the field value under each field in each of the integrated data tables is parsed into a preset mapping value; and different preset mapping values are used to indicate the queue names of different storage queues in the specified storage system.

[0135] The field and the field value are stored in the data storage queue corresponding to the queue name indicated by the preset mapping value.

[0136] In an example embodiment of the present disclosure, the preset parsing rule is used to parse the field values contained in the same field into the same preset mapping value.

[0137] The specific details of each module in the above data table storage device have been described in detail in the corresponding data table storage method, and thus will not be described here.

[0138] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into several modules or units embodied.

[0139] In addition, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0140] Those skilled in the art can clearly understand that the example embodiments described in the present disclosure can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (such as a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0141] The present application also provides a computer-readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device.

[0142] The computer-readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device.

[0143] The computer-readable storage medium can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination of the above.

[0144] The computer-readable storage medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to implement the methods described in the above embodiments.

[0145] In addition, the present disclosure also provides an electronic device capable of implementing the above methods.

[0146] Those skilled in the art can understand that the various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, the various aspects of the present disclosure can be embodied as a whole hardware implementation, a whole software implementation (including firmware, microcode, etc.), or an implementation combined with both hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0147] The electronic device 800 according to this embodiment of the present disclosure will be described below with reference to Figure 8 Figure 8 The electronic device 800 shown is merely one example and should not be taken as limiting the scope of the present disclosure embodiments.

[0148] As shown in Figure 8 , the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 can include, but are not limited to, the at least one processing unit 810 described above, the at least one storage unit 820 described above, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0149] The storage unit stores program code that can be executed by the processing unit 810, so that the processing unit 810 performs the steps described in the "Exemplary Method" section above according to various exemplary embodiments of the present disclosure. For example, the processing unit 810 can perform the steps shown in Figure 1 , step S110, reading information of M data tables to be compressed from a pre-set compression configuration file; M is an integer greater than 1; step S120, obtaining the M data tables according to the information of the M data tables to be compressed; step S130, performing compression and merging on the M data tables according to compression and merging requirement information contained in the compression configuration file, obtaining a compression and merging result; the compression and merging requirement information contains a business primary key corresponding to each data table in the M data tables and at least one field to be compressed corresponding to each data table; step S140, re-compressing the compression and merging result, and storing the obtained re-compression result to a designated storage system.

[0150] The storage unit 820 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 8201 and / or a cache memory unit 8202, and can further include a read-only memory (ROM) 8203.

[0151] ​The storage unit 820 also can include a program / utility 8204 having a set of programs / modules 8205, each of which performs a particular operation or implements a particular abstract data type. These programs / modules 8205 include, but are not limited to, one or more of: one or more application programs, one or more program modules, and program data, each of which operates in conjunction with the operating system 8202. Furthermore, these programs / modules 8205 can include implementations of a networking environment.

[0152] Bus 830 can represent one or more of several types of bus structures, including a storage bus or bus controller, peripheral bus, graphics bus, processor or local bus using any of a variety of bus architectures.

[0153] Electronic device 800 also can communicate with one or more external devices 900 such as a keyboard or pointing device, using one or more communication ports 850. External devices 900 can also include devices that enable a user to interact with electronic device 800 in a manner not directly through use of the keyboard or pointing device. In one embodiment, communication port 850 enables communication between electronic device 800 and one or more devices that enable a user to interact with electronic device 800 in a manner not directly through use of the keyboard or pointing device. In one embodiment, communication port 850 enables communication between electronic device 800 and one or more devices that enable the electronic device 800 to communicate with one or more other computing devices. For example, communication port 850 can enable an electronic device 800 to communicate with an appliance to enable electronic device 800 to operate the appliance. Electronic device 800 can also communicate to one or more devices that enable a user to interact with electronic device 800 in a manner not directly through use of the keyboard or pointing device, and / or to one or more devices that enable the electronic device 800 to communicate with one or more other computing devices, such as a router, a modem, etc. Such communication can occur via Input / Output (I / O) interface 850. Still yet, electronic device 800 can communicate to one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet through network adapter 860. As depicted, network adapter 860 communicates with the other components of electronic device 800 via bus 830. It should be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with electronic device 800. Such as, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0154] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A data table storage method characterized by comprising: The method comprises the following steps: reading information of M data tables to be compressed from a pre-set compression configuration file; M is an integer greater than 1; obtaining the M data tables according to the information of the M data tables to be compressed; performing compression and merging on the M data tables according to compression and merging requirement information contained in the compression configuration file to obtain a compression and merging result; the compression and merging requirement information contains a business primary key corresponding to each data table in the M data tables and at least one field to be compressed corresponding to each data table; performing re-compression on the compression and merging result and storing the obtained re-compression result to a specified storage system.

2. The method of claim 1, wherein, The step of performing compression and merging on the M data tables according to the compression and merging requirement information contained in the compression configuration file to obtain a compression and merging result comprises the following steps: performing field value deduplication and merging on field values under each field to be compressed of each data table to obtain a target data table; selecting at least one data table set with the same business primary key from the M target data tables, each data table set containing at least two target data tables; performing data deduplication and merging on the at least two target data tables to merge the at least two target data tables into one integrated data table; determining at least one integrated data table corresponding to the at least one data table set as the compression and merging result.

3. The method of claim 2, wherein, The step of performing data deduplication and merging on the at least two target data tables to merge the at least two target data tables into one integrated data table comprises the following steps: determining the business primary key corresponding to each data table set as the business primary key corresponding to the integrated data table; performing field deduplication and merging on at least two fields to be compressed corresponding to the at least two target data tables and performing field value deduplication and merging on field values corresponding to the at least two fields to be compressed to merge the at least two target data tables into one integrated data table.

4. The method of claim 2, wherein, The step of performing data deduplication and merging on the at least two target data tables to merge the at least two target data tables into one integrated data table comprises the following steps: determining a joint business primary key configured according to a business requirement as the business primary key corresponding to the integrated data table, the joint business primary key being selected from at least two fields to be compressed corresponding to the at least two target data tables; performing field deduplication and merging on the at least two fields to be compressed corresponding to the at least two target data tables and performing field value deduplication and merging on field values corresponding to the at least two fields to be compressed to merge the at least two target data tables into one integrated data table.

5. The method according to claim 3 or 4, characterized in that, The step of performing re-compression on the compression and merging result comprises the following step: performing data filtering processing on invalid data in the compression and merging result according to a pre-set data filtering rule.

6. The method of claim 5, wherein, The specified storage system contains a plurality of data storage queues, different data storage queues corresponding to different queue names; The step of storing the obtained re-compression result to the specified storage system comprises the following steps: According to a preset analysis rule, field values under each field in each of the integrated data table are analyzed into preset mapping values; different preset mapping values are used to indicate queue names of different storage queues in the specified storage system; The field and the field value are stored in a data storage queue corresponding to the queue name indicated by the preset mapping value.

7. The method of claim 6, wherein, The preset analysis rule is used to analyze field values contained in a same field into a same preset mapping value.

8. A data table storage device, characterized by Comprise: An information reading module, configured to read information of M data tables needing compression from a preset compression configuration file; M is an integer greater than 1; A data table obtaining module, configured to obtain the M data tables according to the information of the M data tables needing compression; A compression and merging module, configured to perform compression and merging on the M data tables according to compression and merging requirement information contained in the compression configuration file, and obtain a compression and merging result; The compression and merging requirement information contains a business primary key corresponding to each of the M data tables and at least one field to be compressed corresponding to each of the M data tables; A re-compression module, configured to perform re-compression on the compression and merging result, and store a re-compression result obtained into a specified storage system.

9. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to implement the data table storage method in any one of claims 1-7.

10. An electronic device, comprising: Comprise: A processor; And A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the executable instructions to implement the data table storage method in any one of claims 1-7.

Images