Data processing method and device

Abstract

The present application discloses a data processing method and device. The method includes: obtaining a data table to be processed; selecting a first number of data columns from the data table to be processed based on the factors affecting the storage space occupied by the data columns in the data table to be processed; selecting a second number of data columns from the data table to be processed based on the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks; and adjusting the arrangement order of the second number of data columns in the data table to be processed to obtain a new data table, wherein the storage space occupied by the new data table is less than the storage space occupied by the data table to be processed. Using the method provided by the present application, the storage space of the data table is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method and device. Background Art

[0002] With the rapid development of data services, data service platforms need to store more and more data. Storing this data requires massive amounts of storage media, which incurs high costs.

[0003] In the prior art, data tables stored in storage media are arranged in chronological order of data generation. To reduce the storage space of the data table, the positions of the data columns are generally kept unchanged before being stored in the storage medium, and only the data in each data column is compressed.

[0004] However, with this data processing method, even if there is similar data between data columns that are relatively far apart, the similar data cannot be uniformly compressed, resulting in an insufficiently significant effect of reducing the storage space of the data table after processing. Summary of the Invention

[0005] The present application provides a data processing method to reduce the storage space of a data table.

[0006] The method comprises:

[0007] Get the data table to be processed;

[0008] selecting a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data table to be processed by the data columns in the data table to be processed;

[0009] selecting a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks;

[0010] The arrangement order of the second number of data columns in the data table to be processed is adjusted to obtain a new data table, where the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0011] Optionally, obtaining the table of data to be processed includes:

[0012] Obtaining the storage space occupied by the data table stored in the non-volatile storage medium;

[0013] According to the size of the storage space occupied by the data table, the to-be-processed data table requiring compressed storage space is obtained from the non-volatile storage medium.

[0014] Optionally, the factors affecting the size of the storage space occupied by the data columns in the data table to be processed include the average field length of the data columns in the data table to be processed;

[0015] The selecting a first number of data columns from the data table to be processed according to the factors affecting the storage size of the data table to be processed by the data columns in the data table to be processed includes:

[0016] A first number of data columns with average field lengths greater than a specified field length threshold are selected from the to-be-processed data table.

[0017] Optionally, the data processing method further includes:

[0018] Obtaining target data blocks in the first number of data columns;

[0019] The number of target data blocks in the first number of data columns is calculated.

[0020] Optionally, obtaining the target data blocks in the first number of data columns includes:

[0021] performing word segmentation processing on the data blocks in the first number of data columns to obtain word segmentation results;

[0022] Obtaining a feature vector of the data block according to the word segmentation result;

[0023] According to the feature vectors in any two data blocks, the distance between the two data blocks is obtained:

[0024] determining, based on a distance between the two arbitrary data blocks, whether the two arbitrary data blocks are similar to each other;

[0025] If the arbitrary two data blocks are similar to each other, the arbitrary two data blocks are determined as the target data blocks.

[0026] Optionally, obtaining a feature vector of a data block according to the word segmentation result includes:

[0027] According to the word segmentation result, the number of words contained in the data block after word segmentation is obtained;

[0028] generating a word dictionary after word segmentation according to the number of words contained in the word segmentation data block;

[0029] According to the word dictionary after word segmentation, a feature vector of the data block is obtained.

[0030] Optionally, obtaining the distance between any two data blocks according to the feature vectors of the any two data blocks includes:

[0031] According to the feature vectors of any two data blocks, obtain the Euclidean distance between any two data blocks or the cosine distance between any two data blocks;

[0032] The distance between the arbitrary two data blocks is obtained according to the Euclidean distance between the arbitrary two data blocks or the cosine distance between the arbitrary two data blocks.

[0033] Optionally, judging whether the arbitrary two data blocks are similar to each other based on the distance between the arbitrary two data blocks includes:

[0034] If the distance between the arbitrary two data blocks is less than a specified distance threshold, it is determined that the arbitrary two data blocks are similar data blocks.

[0035] Optionally, in the new data table, any data column in the second number of data columns is adjacent to another data column in the second number of data columns.

[0036] Optionally, adjusting the arrangement order of the second number of data columns in the to-be-processed data table to obtain a new data table includes:

[0037] In the data table to be processed, setting different arrangement orders between the second number of data columns to obtain multiple new data tables;

[0038] The method further comprises:

[0039] Selecting a new data table with the smallest storage space occupied from the multiple new data tables;

[0040] The new data table occupying the smallest storage space is stored in a storage medium.

[0041] Optionally, the data processing method further includes: determining whether the storage space saved by the new data table occupying the smallest storage space relative to the data table to be processed reaches or exceeds a storage space size threshold;

[0042] Storing the new data table that occupies the smallest storage space in the storage medium includes: if the storage space saved by the new data table that occupies the smallest storage space relative to the data table to be processed is lower than a storage space size threshold, storing the new data table that occupies the smallest storage space in the storage medium.

[0043] Optionally, the data processing method further includes: if the storage space saved by the new data table occupying the smallest storage space relative to the data table to be processed reaches or exceeds the storage space size threshold, then maintaining the arrangement order between the data columns in the data table to be processed.

[0044] The present application provides a data processing device, comprising:

[0045] An acquisition unit, used for acquiring a data table to be processed;

[0046] A first selection unit is configured to select a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data column in the data table to be processed;

[0047] a second selection unit configured to select a second number of data columns from the first number of data columns according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks to each other;

[0048] The adjusting unit is configured to adjust the arrangement order of the second number of data columns in the data table to be processed to obtain a new data table, wherein the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0049] Optionally, the acquiring unit is specifically configured to:

[0050] Obtaining the storage space occupied by the data table stored in the non-volatile storage medium;

[0051] According to the size of the storage space occupied by the data table, the to-be-processed data table requiring compressed storage space is obtained from the non-volatile storage medium.

[0052] Optionally, the first selection unit is specifically configured to:

[0053] A first number of data columns with average field lengths greater than a specified field length threshold are selected from the to-be-processed data table.

[0054] Optionally, the data processing device further includes a computing unit, wherein the computing unit is configured to:

[0055] Obtaining target data blocks in the first number of data columns;

[0056] The number of target data blocks in the first number of data columns is calculated.

[0057] Optionally, the second selection unit is specifically configured to:

[0058] performing word segmentation processing on the data blocks in the first number of data columns to obtain word segmentation results;

[0059] Obtaining a feature vector of the data block according to the word segmentation result;

[0060] Obtaining the distance between any two data blocks according to the feature vectors of the two data blocks;

[0061] determining, based on a distance between the two arbitrary data blocks, whether the two arbitrary data blocks are similar to each other;

[0062] If the arbitrary two data blocks are similar to each other, the arbitrary two data blocks are determined as the target data blocks.

[0063] Optionally, the second selection unit is further configured to:

[0064] According to the word segmentation result, the number of words contained in the data block after word segmentation is obtained;

[0065] generating a word dictionary after word segmentation according to the number of words contained in the word segmentation data block;

[0066] According to the word dictionary after word segmentation, a feature vector of the data block is obtained.

[0067] Optionally, the second selection unit is further configured to:

[0068] According to the feature vectors of any two data blocks, obtain the Euclidean distance between any two data blocks or the cosine distance between any two data blocks;

[0069] The distance between the arbitrary two data blocks is obtained according to the Euclidean distance between the arbitrary two data blocks or the cosine distance between the arbitrary two data blocks.

[0070] Optionally, the second selection unit is further configured to:

[0071] If the distance between the arbitrary two data blocks is less than a specified distance threshold, it is determined that the arbitrary two data blocks are similar data blocks.

[0072] Optionally, in the new data table, any data column in the second number of data columns is adjacent to another data column in the second number of data columns.

[0073] Optionally, the adjustment unit is specifically configured to:

[0074] In the data table to be processed, setting different arrangement orders between the second number of data columns to obtain multiple new data tables;

[0075] The data processing device further includes a storage unit, which is configured to:

[0076] Selecting a new data table with the smallest storage space occupied from the multiple new data tables;

[0077] The new data table occupying the smallest storage space is stored in a storage medium.

[0078] Optionally, the data processing device further includes a judgment unit, configured to judge whether the storage space saved by the new data table with the smallest storage space compared to the data table to be processed reaches or exceeds a storage space size threshold;

[0079] The storage unit is further configured to: store the new data table with the smallest storage space occupied in a storage medium if the storage space saved by the new data table with the smallest storage space occupied relative to the data table to be processed is lower than a storage space size threshold.

[0080] Optionally, the storage unit is further used to: if the storage space saved by the new data table occupying the smallest storage space relative to the data table to be processed reaches or exceeds the storage space size threshold, maintain the arrangement order between the data columns in the data table to be processed.

[0081] This application provides a data processing method, including:

[0082] Get the data table to be processed;

[0083] selecting a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data table to be processed by the data columns in the data table to be processed;

[0084] selecting a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks;

[0085] Selecting a third number of data columns having time attributes from the first number of data columns;

[0086] In the data table to be processed, the arrangement order of the data columns of the second number and the third number is adjusted to obtain a new data table, where the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0087] Optionally, selecting a third number of data columns having time attributes from the first number of data columns includes:

[0088] According to a specified time regular expression, a third number of data columns having time attributes are selected from the first number of data columns.

[0089] The present application provides a data processing device, comprising:

[0090] A first acquiring unit, configured to acquire a data table to be processed;

[0091] a third selecting unit, configured to select a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data column in the data table to be processed;

[0092] a fourth selection unit, configured to select a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks to each other;

[0093] a fifth selecting unit, configured to select a third number of data columns having a time attribute from the first number of data columns;

[0094] The second adjustment unit is used to adjust the arrangement order of the data columns in the data table to be processed for the second number of data columns and the third number of data columns to obtain a new data table, where the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0095] Optionally, the fifth selection unit is specifically configured to:

[0096] According to a specified time regular expression, a third number of data columns having time attributes are selected from the first number of data columns.

[0097] The present application provides an electronic device, including:

[0098] processor;

[0099] as well as.

[0100] The memory is used to store a computer program. After the device runs the computer program through the processor, it executes any one of the data processing methods described above.

[0101] The present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the program is run by a processor to perform any one of the data processing methods described above.

[0102] This application provides a data processing method, including:

[0103] Obtaining a data table A to be processed, wherein A includes multiple data columns, each data column includes at least one data block, and at least one data column corresponds to an impact factor, wherein the impact factor represents an impact factor of a data column on the amount of storage space occupied by A;

[0104] Selecting from A a plurality of data columns whose impact factors meet a first preset condition;

[0105] Based on the similarities of the data blocks included in the plurality of data columns that meet the first preset condition, adjusting the order of the data columns whose similarities meet the second preset condition, and generating a new data table;

[0106] The new data table, when compressed and stored, occupies a smaller storage space than the storage space occupied by A.

[0107] This application provides a data processing method, including:

[0108] Obtaining a data table A to be processed, wherein A includes multiple data columns, and each data column includes at least one data block;

[0109] Selecting a plurality of data columns that meet a third preset condition from A;

[0110] Based on the similarities of the data blocks included in the plurality of data columns that meet the third preset condition, adjusting the order of the data columns whose similarities meet the fourth preset condition, and generating a new data table;

[0111] The new data table, when compressed and stored, occupies a smaller storage space than the storage space occupied by A.

[0112] Compared with the prior art, this application has the following advantages:

[0113] The data processing method provided by the present application obtains a data table to be processed; selects a first number of data columns from the data table to be processed based on the factors affecting the size of the storage space occupied by the data columns in the data table to be processed; selects a second number of data columns from the data table to be processed based on the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks; and adjusts the arrangement order between the second number of data columns in the data table to be processed to obtain a new data table, wherein the storage space occupied by the new data table is less than the storage space occupied by the data table to be processed. Using the method provided by the present application, by selecting data columns that affect the storage space, and then analyzing whether these data columns that affect the storage size are similar, and by adjusting the arrangement order between the data columns, the storage space of the data table can be significantly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 This is a schematic diagram of an application scenario embodiment of a data processing method provided by this application;

[0115] Figure 2This is a flow chart of a data processing method provided in the first embodiment of the present application;

[0116] Figure 3 is a schematic diagram of a data processing device provided in the second embodiment of the present application;

[0117] Figure 4 is a flow chart of a data processing method provided in the third embodiment of the present application;

[0118] Figure 5 This is a workflow diagram of an application system involved in the third embodiment of the present application;

[0119] Figure 6 is a schematic diagram of a data processing device provided in a fourth embodiment of the present application;

[0120] Figure 7 is a flow chart of a data processing method provided in the seventh embodiment of the present application;

[0121] Figure 8 This is a flowchart of a data processing method provided in the eighth embodiment of the present application. DETAILED DESCRIPTION

[0122] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0123] In order to enable those skilled in the art to better understand the present application, a specific application scenario embodiment of the present application is first described in detail. Figure 1 As shown, it is a schematic diagram of an embodiment of an application scenario of a data processing method provided by the present application. In a specific implementation process, a client can send a request message for obtaining a data table to be processed to a database server; after obtaining the data table to be processed, the client selects a first number of data columns from the data table to be processed based on the factors affecting the size of the storage space occupied by the data columns in the data table to be processed; based on the number of target data blocks in the first number of data columns, a second number of data columns is selected from the data table to be processed, wherein the target data block and at least one data block in the first number of data columns are similar data blocks to each other; in the data table to be processed, the arrangement order between the second number of data columns is adjusted to obtain a new data table, and the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed; the client sends a request to the database server to store the new data table.

[0124] The first embodiment of the present application provides a data processing method. Figure 2 , which is a flow chart of the first embodiment of this application. Figure 1 A data processing method is provided in detail for the first embodiment of the present application. The method comprises the following steps:

[0125] Step S201: Obtain a table of data to be processed.

[0126] This step is used to obtain the data table to be processed.

[0127] A data table is a structured data file used to store specific types of data. To reduce storage space when storing data columns on non-volatile storage media, data tables are typically compressed. Data tables with the same content, but with different data distributions, will achieve different compression ratios when stored on non-volatile storage media. This is because the distribution of similar data blocks affects the compression ratio during compression.

[0128] With the rapid rise of cloud computing, a large number of data tables have been migrated to the cloud. Generally speaking, cloud platforms charge customers based on their storage capacity. Therefore, in order to reduce storage overhead, reducing the storage space of data tables has become an urgent need.

[0129] In cloud data platforms commonly used in the industry, data tables are collections of data. This means that tables with different data distributions are equivalent to each other. However, in actual storage, different data distributions can significantly differ in their suitability for compression algorithms. For example, sorting a table by different columns can result in different storage spaces.

[0130] Generally speaking, every data table has one or more fields that significantly impact storage space. These fields are key factors affecting compression effectiveness. Among them, average field length and the number of unique values ​​are two extremely critical reference values.

[0131] The step of obtaining the data table to be processed includes:

[0132] Obtaining the storage space occupied by the data table stored in the non-volatile storage medium;

[0133] According to the size of the storage space occupied by the data table, the to-be-processed data table requiring compressed storage space is obtained from the non-volatile storage medium.

[0134] To quickly reduce a customer's cloud storage space, it's necessary to quickly identify data tables that consume significant storage space. First, the storage space occupied by the data tables stored in the cloud's non-volatile storage medium is obtained. Then, the data tables are sorted by the space occupied, and the data tables that occupy the most space are retrieved from the non-volatile storage medium as the data tables to be processed.

[0135] Step S202: selecting a first number of data columns from the data table to be processed according to factors affecting the size of storage space occupied by the data columns in the data table to be processed.

[0136] This step is used to select a first number of data columns from the data table to be processed according to the factors affecting the storage space occupied by the data table to be processed by the data columns in the data table to be processed.

[0137] In this embodiment, selecting the first number of data columns may be selecting the first three data columns among the data columns that most affect the storage space of the data table.

[0138] The factors affecting the amount of storage space occupied by the data columns in the data table to be processed include the average field length of the data columns in the data table to be processed;

[0139] The selecting a first number of data columns from the data table to be processed according to the factors affecting the storage size of the data table to be processed by the data columns in the data table to be processed includes:

[0140] A first number of data columns with average field lengths greater than a specified field length threshold are selected from the to-be-processed data table.

[0141] For example, the field length threshold may be set to X bytes, and then Y data columns with an average field length greater than X are selected from the to-be-processed data table.

[0142] Step S203: selecting a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks.

[0143] This step is used to select a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks.

[0144] For example, the first number of data columns may be the first three data columns that most affect the storage space of the data table. The following operations are performed on the target data blocks of the first number of data columns {A, B, C} (the set of data columns other than the first number is {a, b, c, ..., n}):

[0145] Select

[0146] Count(distinct non-first number of data columns a) as a_cnt,

[0147] Count(distinct non-first number of data columns b) as b_cnt,

[0148] Count(distinct non-first number of data columns c) as c_cnt,...

[0149] Count(distinct non-first number of data columns n) as n_cnt

[0150] From Table t group by the first number of data columns_A target data block;

[0151] Select the column with the minimum value of the top2 from a_cnt, b_cnt, c_cnt, ..., n_cnt. For example, {non-first number of data columns a, non-first number of data columns b} is the key set {a, b} that has the greatest impact on the first number of data columns A.

[0152] Select

[0153] Count(distinct non-first number of data columns a) as a_cnt,

[0154] Count(distinct non-first number of data columns b) as b_cnt,

[0155] Count(distinct non-first number of data columns c) as c_cnt,...

[0156] Count(distinct non-first number of data columns n) as n_cnt

[0157] From Table t group by the first number of data columns _B target data blocks;

[0158] Select the column with the minimum value of the top2 from a_cnt, b_cnt, c_cnt, ..., n_cnt, for example, {non-first number of data columns a, non-first number of data columns b}, which is the key set {a, c} that has the greatest impact on the first number of data columns B.

[0159] The data processing method further includes:

[0160] Obtaining target data blocks in the first number of data columns;

[0161] The number of target data blocks in the first number of data columns is calculated.

[0162] The above steps are described in detail below.

[0163] The obtaining of the target data blocks in the first number of data columns includes:

[0164] performing word segmentation processing on the data blocks in the first number of data columns to obtain word segmentation results;

[0165] Obtaining a feature vector of the data block according to the word segmentation result;

[0166] Obtaining the distance between any two data blocks according to the feature vectors of the two data blocks;

[0167] determining, based on a distance between the two arbitrary data blocks, whether the two arbitrary data blocks are similar to each other;

[0168] If the arbitrary two data blocks are similar to each other, the arbitrary two data blocks are determined as the target data blocks.

[0169] First, the data blocks in the first number of data columns are segmented according to a specific format (such as spaces) to obtain a segmentation result; then, based on the number of all words contained after the segmentation, a word dictionary is formed, and each word corresponds to a vector element. According to the data blocks in the first number of data columns, the feature vectors of any two data blocks are obtained by one-hot encoding; then, based on the feature vectors of any two data blocks, the distance between the any two data blocks is obtained; then, based on the distance between the any two data blocks, whether the any two data blocks are similar to each other is determined, for example, whether the distance between the any two data blocks is less than a pre-set distance threshold. If so, the any two data blocks are determined to be similar to each other; finally, if the any two data blocks are similar to each other, the any two data blocks are determined as the target data blocks.

[0170] Obtaining a feature vector of a data block according to the word segmentation result includes:

[0171] According to the word segmentation result, the number of words contained in the data block after word segmentation is obtained;

[0172] generating a word dictionary after word segmentation according to the number of words contained in the word segmentation data block;

[0173] According to the word dictionary after word segmentation, a feature vector of the data block is obtained.

[0174] For example, there are three text blocks as follows:

[0175] 1: I love China;

[0176] 2: Mom and Dad love me;

[0177] 3: Mom and Dad love China;

[0178] After word segmentation, the following dictionary is generated:

[0179] 1: Me 2: Love 3: Dad 4: Mom 5: China;

[0180] Using one-hot feature extraction for the three text blocks, the obtained feature vectors are:

[0181] I love China: [1, 1, 0, 0, 1];

[0182] Mom and Dad love me: [1, 1, 1, 1, 0];

[0183] Mom and Dad love China: [0, 1, 1, 1, 1];

[0184] The obtaining the distance between any two data blocks according to the feature vectors of the any two data blocks includes:

[0185] According to the feature vectors of any two data blocks, obtain the Euclidean distance between any two data blocks or the cosine distance between any two data blocks;

[0186] The distance between the arbitrary two data blocks is obtained according to the Euclidean distance between the arbitrary two data blocks or the cosine distance between the arbitrary two data blocks.

[0187] For example, for two vectors X and Y, the calculation formula for the cosine distance dist(X, Y) is:

[0188]

[0189] The calculation formula of its Euclidean distance sim(X, Y) is:

[0190]

[0191] Using the above formula, the Euclidean distance or the cosine distance between any two data blocks can be obtained. The Euclidean distance or the cosine distance between any two data blocks can be used as the distance between any two data blocks.

[0192] Since calculating Euclidean distance and cosine distance are commonly used calculation steps, they are not given as examples here.

[0193] The determining, based on the distance between the two arbitrary data blocks, whether the two arbitrary data blocks are similar to each other includes:

[0194] If the distance between the arbitrary two data blocks is less than a specified distance threshold, it is determined that the arbitrary two data blocks are similar data blocks.

[0195] After finding similar data blocks, you can use the following SQL statement to perform statistics:

[0196] select count(col1), count(col2)..., count(coln) from table group by similar data blocks;

[0197] Wherein, col1, col2, col3...coln are data columns in the data table to be processed. After completing the above statistics, the three columns with the smallest statistical values ​​can be found as the second number of data columns.

[0198] In the new data table, any data column in the second number of data columns is adjacent to another data column in the second number of data columns.

[0199] For example, each data column in the second number of data columns may be arranged together.

[0200] Step S204: adjusting the arrangement order of the second number of data columns in the data table to be processed to obtain a new data table, wherein the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0201] This step is used to adjust the arrangement order of the second number of data columns in the data table to be processed to obtain a new data table, where the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0202] The step of adjusting the arrangement order of the second number of data columns in the to-be-processed data table to obtain a new data table includes:

[0203] In the data table to be processed, setting different arrangement orders between the second number of data columns to obtain multiple new data tables;

[0204] The method further comprises:

[0205] Selecting a new data table with the smallest storage space occupied from the multiple new data tables;

[0206] The new data table occupying the smallest storage space is stored in a storage medium.

[0207] For example, if the second number of data columns consists of three data columns A, B, and C, adjusting the order of columns A, B, and C yields six permutations and combinations: ABC, ACB, BAC, BCA, CAB, and CBA. These six permutations and combinations correspond to six new data tables. From these six new data tables, the new data table that occupies the least storage space on the data platform is selected, and then the new data table that occupies the least storage space is stored in the storage medium.

[0208] The data processing method further includes: determining whether the storage space saved by the new data table that occupies the smallest storage space relative to the data table to be processed reaches or exceeds a storage space size threshold;

[0209] Storing the new data table that occupies the smallest storage space in the storage medium includes: if the storage space saved by the new data table that occupies the smallest storage space relative to the data table to be processed reaches or exceeds a storage space size threshold, storing the new data table that occupies the smallest storage space in the storage medium.

[0210] For example, if the storage space size threshold is 1.5 GB, and the storage space occupied by the new data table with the smallest storage space is 1.4 GB, the new data table with the smallest storage space is stored in the storage medium.

[0211] The data processing method further includes: if the storage space saved by the new data table occupying the smallest storage space relative to the data table to be processed is lower than the storage space size threshold, maintaining the arrangement order between the data columns in the data table to be processed.

[0212] For example, if the storage space size threshold is 1.5 GB, and the storage space occupied by the new data table with the smallest storage space is 1.6 GB, the arrangement order of the data columns in the data table to be processed is maintained.

[0213] In the above embodiment, a data processing method is provided. Correspondingly, the present application also provides a data processing device. Figure 3, which is a flow chart of an embodiment of a data processing device of the present application. Since this embodiment, i.e., the second embodiment, is substantially similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description of the method embodiment. The device embodiment described below is merely illustrative.

[0214] The present application provides a data processing device, comprising:

[0215] An acquisition unit 301 is used to acquire a data table to be processed;

[0216] A first selection unit 302 is configured to select a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data column in the data table to be processed;

[0217] A second selection unit 303 is configured to select a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks to each other;

[0218] The adjusting unit 304 is configured to adjust the arrangement order of the second number of data columns in the data table to be processed to obtain a new data table, wherein the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0219] In this embodiment, the acquisition unit is specifically configured to:

[0220] Obtaining the storage space occupied by the data table stored in the non-volatile storage medium;

[0221] According to the size of the storage space occupied by the data table, the to-be-processed data table requiring compressed storage space is obtained from the non-volatile storage medium.

[0222] In this embodiment, the first selection unit is specifically configured to:

[0223] A first number of data columns with average field lengths greater than a specified field length threshold are selected from the to-be-processed data table.

[0224] In this embodiment, the data processing device further includes a computing unit, which is configured to:

[0225] Obtaining target data blocks in the first number of data columns;

[0226] The number of target data blocks in the first number of data columns is calculated.

[0227] In this embodiment, the second selection unit is specifically configured to:

[0228] performing word segmentation processing on the data blocks in the first number of data columns to obtain word segmentation results;

[0229] Obtaining a feature vector of the data block according to the word segmentation result;

[0230] Obtaining the distance between any two data blocks according to the feature vectors of the two data blocks;

[0231] determining, based on a distance between the two arbitrary data blocks, whether the two arbitrary data blocks are similar to each other;

[0232] If the arbitrary two data blocks are similar to each other, the arbitrary two data blocks are determined as the target data blocks.

[0233] In this embodiment, the second selection unit is further configured to:

[0234] According to the word segmentation result, the number of words contained in the data block after word segmentation is obtained;

[0235] generating a word dictionary after word segmentation according to the number of words contained in the word segmentation data block;

[0236] According to the word dictionary after word segmentation, a feature vector of the data block is obtained.

[0237] In this embodiment, the second selection unit is further configured to:

[0238] According to the feature vectors of any two data blocks, obtain the Euclidean distance between any two data blocks or the cosine distance between any two data blocks;

[0239] The distance between the arbitrary two data blocks is obtained according to the Euclidean distance between the arbitrary two data blocks or the cosine distance between the arbitrary two data blocks.

[0240] In this embodiment, the second selection unit is further configured to:

[0241] If the distance between the arbitrary two data blocks is less than a specified distance threshold, it is determined that the arbitrary two data blocks are similar data blocks.

[0242] In this embodiment, in the new data table, any data column in the second number of data columns is adjacent to another data column in the second number of data columns.

[0243] In this embodiment, the adjustment unit is specifically configured to:

[0244] In the data table to be processed, setting different arrangement orders between the second number of data columns to obtain multiple new data tables;

[0245] The data processing device further includes a storage unit, which is configured to:

[0246] Selecting a new data table with the smallest storage space occupied from the multiple new data tables;

[0247] The new data table occupying the smallest storage space is stored in a storage medium.

[0248] In this embodiment, the data processing device further includes a judgment unit, configured to judge whether the storage space saved by the new data table with the smallest storage space compared to the data table to be processed reaches or exceeds a storage space size threshold;

[0249] The storage unit is further configured to: store the new data table with the smallest storage space occupied in a storage medium if the storage space saved by the new data table with the smallest storage space occupied relative to the data table to be processed reaches or exceeds a storage space size threshold.

[0250] In this embodiment, the storage unit is further used to: if the storage space saved by the new data table occupying the smallest storage space relative to the data table to be processed is lower than the storage space size threshold, maintain the arrangement order between the data columns in the data table to be processed.

[0251] The third embodiment of the present application provides a data processing method. Since this embodiment has many overlapping parts with the first embodiment of the present application, the following description will not be repeated for the overlapping parts. Please refer to the relevant parts of the first embodiment. Figure 4 , which is a flow chart of a data processing method provided in this embodiment. The data processing method includes the following steps:

[0252] Step S401: Obtain a table of data to be processed.

[0253] This step is used to obtain the data table to be processed.

[0254] Step S402: selecting a first number of data columns from the data table to be processed according to factors affecting the size of storage space occupied by the data columns in the data table to be processed.

[0255] This step is used to select a first number of data columns from the data table to be processed according to the factors affecting the storage space occupied by the data table to be processed by the data columns in the data table to be processed.

[0256] Step S403: selecting a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks.

[0257] This step is used to select a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks.

[0258] Step S404: selecting a third number of data columns having time attributes from the first number of data columns.

[0259] This step is used to select a third number of data columns having time attributes from the first number of data columns.

[0260] The selecting a third number of data columns having time attributes from the first number of data columns includes:

[0261] According to a specified time regular expression, a third number of data columns having time attributes are selected from the first number of data columns.

[0262] The following is a specific example of implementing a time regular expression using the JAVA language.

[0263]

[0264] Step S405: In the data table to be processed, for the second number of data columns and the third number of data columns, adjusting the order of the data columns to obtain a new data table, wherein the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0265] This step is used to adjust the arrangement order of the data columns in the data table to be processed for the second number of data columns and the third number of data columns to obtain a new data table, where the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0266] For example, if the second number of data columns are A, B, and C, and the third number of data columns are D and E, then the order of the data columns can be adjusted to have five factorial arrangements, such as ABCDEF, ABCDFE, and ABCDEF. Each arrangement corresponds to a new data table.

[0267] Figure 5A workflow diagram of an application system using the data processing method provided in this embodiment is provided. The working steps of the application system include:

[0268] Step S501: Select a data table whose storage space needs to be reduced.

[0269] Step S502: Through data analysis and exploration, select the top data columns that affect the storage space size of the data table.

[0270] Step S503: Automatically analyze the attributes of the top data column related to the data column in step S402 through the algorithm module, which specifically includes the following sub-steps:

[0271] Step S503-1: Using a similar text block algorithm, find similar text blocks in the top column. Specific implementation steps refer to the first embodiment of this application.

[0272] Step S503-2: similar text blocks are clustered by executing the following SQL statement:

[0273] select count(col1), count(co12)..., count(coln) from table group by similar text blocks;

[0274] Wherein, col1, col2, ...coln represent the top data columns respectively.

[0275] Step S503-3: Find several columns with the smallest statistical values ​​as set A.

[0276] Step S503-4: Select the columns representing time using the time regular expression as set B

[0277] S504: Traverse and combine the columns in A and the columns in B, redistribute the data (distribute by al, an sort by al, an, bn), and compare the storage space size of the redistributed data table with the storage space size of the original data table to obtain the results before and after the comparison.

[0278] S505: Select the best combination result. If the storage is significantly reduced (for example, greater than 20%), re-sort the data according to the data column to reduce the storage size. Otherwise, the data table is not suitable for reducing the storage in this way.

[0279] In the above embodiment, a data processing method is provided. Correspondingly, the present application also provides a data processing device. Figure 6, which is a flow chart of an embodiment of a data processing device of the present application. Since this embodiment, namely the fourth embodiment, is substantially similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment. The device embodiment described below is merely illustrative.

[0280] The present application provides a data processing device, comprising:

[0281] A first acquiring unit 601 is used to acquire a data table to be processed;

[0282] A third selection unit 602 is configured to select a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data column in the data table to be processed;

[0283] a fourth selection unit 603 configured to select a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks to each other;

[0284] a fifth selecting unit 604, configured to select a third number of data columns having a time attribute from the first number of data columns;

[0285] The second adjusting unit 605 is configured to adjust the arrangement order of the data columns in the data table to be processed, with respect to the second number of data columns and the third number of data columns, to obtain a new data table, wherein the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

[0286] Optionally, the fifth selection unit is specifically configured to:

[0287] According to a specified time regular expression, a third number of data columns having time attributes are selected from the first number of data columns.

[0288] A fifth embodiment of the present application provides an electronic device, including:

[0289] processor;

[0290] as well as.

[0291] The memory is used to store a computer program. After the device runs the computer program through the processor, it executes any one of the data processing methods provided in the first embodiment and the third embodiment of the present application.

[0292] The sixth embodiment of the present application provides a computer storage medium, which stores a computer program. The program is run by a processor to execute any one of the data processing methods provided in the first embodiment and the third embodiment of the present application.

[0293] The seventh embodiment of this application provides a data processing method, please refer to Figure 7 , which is a flow chart of a data processing method provided in the seventh embodiment of the present application. Since this embodiment is similar to the first and third embodiments of the present application, only a brief description is given here. The method includes the following steps:

[0294] Step S701: obtaining a data table A to be processed, wherein A includes multiple data columns, one data column includes at least one data block, and at least one data column corresponds to an impact factor, which represents the impact factor of a data column on the size of the storage space occupied by A.

[0295] This step is used to obtain a data table A to be processed, wherein A includes multiple data columns, each data column includes at least one data block, and at least one data column corresponds to an impact factor, which represents the impact factor of a data column on the size of the storage space occupied by A.

[0296] The factors affecting the size of the storage space occupied by A by a data column include the average field length of the data column.

[0297] Step S702: Selecting from A a plurality of data columns whose impact factors meet a first preset condition.

[0298] This step is used to select a plurality of data columns whose impact factors meet the first preset condition from A.

[0299] For example, the first three data columns that most affect the storage space of the data table are selected.

[0300] Step S703: Based on the similarity of the data blocks included in the multiple data columns that meet the first preset condition, adjust the order of the data columns whose similarity meets the second preset condition to generate a new data table; wherein, when compressed and stored, the new data table occupies a storage space that is smaller than the storage space occupied by A.

[0301] For example, the cosine distance or Euclidean distance between the data blocks included in the plurality of data columns is calculated to obtain the similarity of the data blocks included in the plurality of data columns. The order of the data columns whose similarities meet the second preset condition is adjusted to generate a new data table.

[0302] The eighth embodiment of this application provides a data processing method, please refer to Figure 8, which is a flow chart of a data processing method provided in the seventh embodiment of the present application. Since this embodiment is similar to the first and third embodiments of the present application, only a brief description is given here. The data processing method includes:

[0303] Step S801: obtaining a data table A to be processed, wherein A includes multiple data columns, and each data column includes at least one data block.

[0304] This step is used to obtain a data table A to be processed, wherein A includes multiple data columns, and each data column includes at least one data block.

[0305] Step S802: Select multiple data columns that meet a third preset condition from A.

[0306] This step is used to select multiple data columns that meet the third preset condition from A.

[0307] Step S803: Based on the similarity of the data blocks included in the multiple data columns that meet the third preset condition, adjust the order of the data columns whose similarity meets the fourth preset condition to generate a new data table; wherein, when compressed and stored, the new data table occupies a storage space that is smaller than the storage space occupied by A.

[0308] This step is used to adjust the order of data columns whose similarities meet the fourth preset condition based on the similarities of the data blocks included in the multiple data columns, and generate a new data table; wherein, when compressed and stored, the new data table occupies a storage space that is smaller than the storage space occupied by A.

[0309] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0310] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0311] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0312] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0313] 2. Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A data processing method, characterized in that: include: Get the data table to be processed; selecting a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data table to be processed by the data columns in the data table to be processed; selecting a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks; wherein, after determining the similar data blocks by performing statistics through SQL statements, after completing statistics on the data table to be processed, selecting data columns with the smallest statistical value from the data table to be processed as the second number of data columns, the statistical value being the number of non-null values; The arrangement order of the second number of data columns in the data table to be processed is adjusted to obtain a new data table, where the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

2. The data processing method according to claim 1, wherein: The step of obtaining the data table to be processed includes: Obtaining the storage space occupied by the data table stored in the non-volatile storage medium; According to the size of the storage space occupied by the data table, the to-be-processed data table requiring compressed storage space is obtained from the non-volatile storage medium.

3. The data processing method according to claim 1, wherein: The factors affecting the amount of storage space occupied by the data columns in the data table to be processed include the average field length of the data columns in the data table to be processed; The selecting a first number of data columns from the data table to be processed according to the factors affecting the storage size of the data table to be processed by the data columns in the data table to be processed includes: A first number of data columns with average field lengths greater than a specified field length threshold are selected from the to-be-processed data table.

4. The data processing method according to claim 1, wherein: Also includes: Obtaining target data blocks in the first number of data columns; The number of target data blocks in the first number of data columns is calculated.

5. The data processing method according to claim 4, characterized in that: The obtaining of the target data blocks in the first number of data columns includes: performing word segmentation processing on the data blocks in the first number of data columns to obtain word segmentation results; Obtaining a feature vector of the data block according to the word segmentation result; Obtaining the distance between any two data blocks according to the feature vectors of the two data blocks; determining whether the arbitrary two data blocks are similar to each other according to the distance between the arbitrary two data blocks, and determining the arbitrary two data blocks as the target data blocks; The determining whether the arbitrary two data blocks are similar to each other based on the distance between the arbitrary two data blocks includes: if the distance between the arbitrary two data blocks is less than a specified distance threshold, determining that the arbitrary two data blocks are similar to each other.

6. The data processing method according to claim 5, characterized in that: Obtaining a feature vector of a data block according to the word segmentation result includes: According to the word segmentation result, the number of words contained in the data block after word segmentation is obtained; generating a word dictionary after word segmentation according to the number of words contained in the word segmentation data block; According to the word dictionary after word segmentation, a feature vector of the data block is obtained.

7. The data processing method according to claim 5, characterized in that: The obtaining the distance between any two data blocks according to the feature vectors of the any two data blocks includes: According to the feature vectors of any two data blocks, obtain the Euclidean distance between any two data blocks or the cosine distance between any two data blocks; The distance between the arbitrary two data blocks is obtained according to the Euclidean distance between the arbitrary two data blocks or the cosine distance between the arbitrary two data blocks.

8. The data processing method according to claim 1, wherein: In the new data table, any data column in the second number of data columns is adjacent to another data column in the second number of data columns.

9. The data processing method according to claim 1, wherein: The step of adjusting the arrangement order of the second number of data columns in the to-be-processed data table to obtain a new data table includes: In the data table to be processed, setting different arrangement orders between the second number of data columns to obtain multiple new data tables; The method further comprises: Selecting a new data table with the smallest storage space occupied from the multiple new data tables; The new data table occupying the smallest storage space is stored in a storage medium.

10. The data processing method according to claim 9, characterized in that: Also includes: Determining whether the storage space saved by the new data table that occupies the smallest storage space relative to the data table to be processed reaches or exceeds a storage space size threshold; Storing the new data table that occupies the smallest storage space in the storage medium includes: if the storage space saved by the new data table that occupies the smallest storage space relative to the data table to be processed is lower than a storage space size threshold, storing the new data table that occupies the smallest storage space in the storage medium.

11. The data processing method according to claim 10, characterized in that: Also includes: If the storage space saved by the new data table occupying the smallest storage space relative to the data table to be processed reaches or exceeds the storage space size threshold, the arrangement order of the data columns in the data table to be processed is maintained.

12. A data processing device, characterized in that: include: An acquisition unit, used for acquiring a data table to be processed; A first selection unit is configured to select a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data column in the data table to be processed; a second selection unit, configured to select a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks to each other; wherein, after determining the similar data blocks by performing statistics through SQL statements, after completing statistics on the data table to be processed, selecting data columns with the smallest statistical value from the data table to be processed as the second number of data columns, the statistical value being the number of non-null values; The adjusting unit is configured to adjust the arrangement order of the second number of data columns in the data table to be processed to obtain a new data table, wherein the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

13. A data processing method, characterized in that: include: Get the data table to be processed; selecting a first number of data columns from the data table to be processed according to factors affecting the amount of storage space occupied by the data table to be processed by the data columns in the data table to be processed; selecting a second number of data columns from the to-be-processed data table according to the number of target data blocks in the first number of data columns, wherein the target data block and at least one data block in the first number of data columns are similar data blocks; wherein, after determining the similar data blocks by performing statistics through SQL statements, after completing statistics on the data table to be processed, selecting data columns with the smallest statistical value from the data table to be processed as the second number of data columns, the statistical value being the number of non-null values; Selecting a third number of data columns having time attributes from the first number of data columns; In the data table to be processed, the arrangement order of the data columns of the second number and the third number is adjusted to obtain a new data table, where the storage space occupied by the new data table is smaller than the storage space occupied by the data table to be processed.

14. The data processing method according to claim 13, wherein: The selecting a third number of data columns having time attributes from the first number of data columns includes: According to a specified time regular expression, a third number of data columns having time attributes are selected from the first number of data columns.

15. An electronic device, characterized in that: include: processor; as well as, The memory is used to store a computer program. After the device runs the computer program through the processor, it executes the method according to any one of claims 1 to 11 and 13 to 14.

16. A computer storage medium, characterized in that The computer storage medium stores a computer program, which is executed by a processor to perform the method according to any one of claims 1 to 11 and 13 to 14.

Images