Data processing method and apparatus, and electronic device

Abstract

Embodiments of the present application relate to the technical field of data processing, and disclose a data processing method, device and electronic equipment. The method comprises: obtaining to-be-processed data, wherein the to-be-processed data comprises data of at least one to-be-written data row; classifying the to-be-processed data to obtain classification information of the to-be-processed data; determining a target row group corresponding to the classification information among a plurality of row groups included in a column storage file according to the classification information of the to-be-processed data, wherein the classification information corresponding to different row groups among the plurality of row groups is different; and writing the to-be-processed data into the target row group in the column storage file. The technical solution of the present application can reduce the occupation of storage space by data and improve the data compression ratio.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, specifically to a data processing method, apparatus, and electronic device. Background Technology

[0002] With the development of big data technology, more and more data are being connected to big data storage systems, providing data support for analyzing business processes and optimizing service quality.

[0003] Currently, storing data for each time period (e.g., daily) in a separate data partition is a relatively reliable method. However, because the data for each time period may contain a lot of unchanging information, the data partition will contain a lot of duplicate data, resulting in a significant consumption of storage space and a waste of storage resources. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a data processing method, apparatus and electronic device to solve the problem of excessive storage space occupied by data in the prior art.

[0005] According to one aspect of the present invention, a data processing method is provided, the method comprising: acquiring data to be processed, the data to be processed including at least one data row to be written; classifying the data to be processed to obtain classification information of the data to be processed; determining a target row group corresponding to the classification information among a plurality of row groups included in a column storage file according to the classification information of the data to be processed; different row groups in the plurality of row groups corresponding to different classification information; and writing the data to be processed into the target row group in the column storage file.

[0006] In one optional approach, before classifying the data to be processed to obtain classification information, the method further includes: determining the encoding information of data in each data row to be written in the data to be processed; determining, in an index file, the encoding information of at least one stored data row with the same data generation time as the data row to be written, based on the data generation time of the data row to be written; the index file includes the encoding information of data in multiple stored data rows in the column storage file; if the encoding information of the data row to be written is different from the encoding information of the stored data rows, then the step of classifying the data to be processed to obtain classification information is performed.

[0007] In an optional embodiment, the method further includes: if the encoding information of the data to be written in the data row is the same as the encoding information of the data in the stored data row, then discarding the data in the data row to be written.

[0008] In an optional embodiment, the index file further includes multiple row group identifiers and identifiers of each stored data row in the column storage file. The method further includes: obtaining a query instruction, the query instruction including the data generation time of the data to be queried; determining, in the index file, a row group identifier and multiple identifiers of stored data rows corresponding to the data generation time of the data to be queried; and determining, in the column storage file, the data to be queried based on the row group identifier and the multiple identifiers of stored data rows.

[0009] In one alternative approach, before classifying the data to be processed to obtain classification information of the data to be processed, the method further includes: comparing the number of data rows to be written included in the data to be processed with a preset number; if the number of data rows to be written included in the data to be processed is greater than or equal to the preset number, then the step of classifying the data to be processed to obtain classification information of the data to be processed is performed.

[0010] In one optional approach, classifying the data to be processed to obtain classification information of the data to be processed includes: classifying the data of the preset number of data rows to be written to obtain classification information of the preset number of data rows to be written; the preset number of data rows to be written is determined based on the data generation time of each data row to be written in the data to be processed.

[0011] In one optional approach, the method further includes: acquiring new data to be processed; if the sum of the number of unclassified rows of data to be written in the data to be processed and the number of new rows of data to be written in the new data to be processed is greater than or equal to the preset number, then the data of the unclassified rows of data to be written and the data of the new rows of data to be written are classified to obtain new classification information.

[0012] In one optional manner, the target row group includes multiple column groups, each of the multiple column groups including a data column, and the step of writing the data to be processed into the target row group in the column storage file includes: storing data of the corresponding data column in the data to be processed in the target row group in the column storage file according to the attribute information of each data column in the data to be processed; the multiple column groups include the target column group.

[0013] According to another aspect of the present invention, a data processing apparatus is provided, the apparatus comprising: an acquisition module for acquiring data to be processed, the data to be processed including at least one data row to be written; a classification module for classifying the data to be processed to obtain classification information of the data to be processed; a determination module for determining a target row group corresponding to the classification information in a plurality of row groups included in a column storage file based on the classification information of the data to be processed; wherein different row groups in the plurality of row groups correspond to different classification information; and a writing module for writing the data to be processed into the target row group in the column storage file.

[0014] In one optional approach, before classifying the data to be processed to obtain classification information, the classification module is further configured to determine the encoding information of each data row to be written in the data to be processed, and, based on the data generation time of the data row to be written, determine the encoding information of at least one stored data row with the same data generation time as the data row to be written in the index file, wherein the index file includes the encoding information of multiple stored data rows in the column storage file. If the encoding information of the data row to be written is different from the encoding information of the stored data rows, then the step of classifying the data to be processed to obtain classification information is performed.

[0015] In an alternative embodiment, the classification module is further configured to discard the data in the data row to be written if the encoding information of the data in the data row to be written is the same as the encoding information of the data in the stored data row.

[0016] In one optional embodiment, the index file further includes multiple row group identifiers and identifiers of each stored data row in the column storage file. The writing module is further configured to obtain a query instruction, the query instruction including the data generation time of the data to be queried, and, based on the data generation time of the data to be queried, determine the row group identifier and the identifiers of multiple stored data rows corresponding to the data generation time of the data to be queried in the index file, and, based on the row group identifier and the identifiers of the multiple stored data rows, determine the data to be queried in the column storage file.

[0017] In one optional approach, before classifying the data to be processed to obtain classification information, the classification module is further configured to compare the number of data rows to be written included in the data to be processed with a preset number. If the number of data rows to be written included in the data to be processed is greater than or equal to the preset number, then the step of classifying the data to be processed to obtain classification information is performed.

[0018] In one optional approach, the classification module is used to classify the data of the preset number of data rows to be written, and obtain classification information of the preset number of data rows to be written; the preset number of data rows to be written is determined according to the data generation time of each data row to be written in the data to be processed.

[0019] In one optional approach, the classification module is further configured to acquire new data to be processed. If the sum of the number of unclassified rows of data to be written in the data to be processed and the number of new rows of data to be written in the new data to be processed is greater than or equal to the preset number, then the data of the unclassified rows of data to be written and the data of the new rows of data to be written are classified to obtain new classification information.

[0020] In one optional manner, the target row group includes multiple column groups, each of the multiple column groups including a data column. The writing module is used to store the data of the corresponding data column in the data to be processed in the target row group in the column storage file according to the attribute information of each data column in the data to be processed, in the target column group corresponding to the attribute information. The multiple column groups include the target column group.

[0021] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing at least one executable instruction; the executable instruction causing the processor to perform operations as described in any of the preceding data processing methods.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, which, when executed on an electronic device, causes the electronic device to perform the operation of the data processing method as described in any of the preceding claims.

[0023] According to the data processing method, apparatus, and electronic device provided in embodiments of the present invention, data to be processed can be acquired, classified to obtain classification information, and then, based on the classification information, a target row group corresponding to the classification information is determined from multiple row groups included in the column storage file. The data to be processed is then written into the target row group in the column storage file. This method allows data to be written column-wise into the column storage file, ensuring that the data is stored in target row groups with similar data characteristics, thereby improving the data compression ratio and saving data storage space. Furthermore, due to the advantages of the column storage file format, when reading stored data, only the data in the corresponding column needs to be read, rather than reading every row, thus improving data reading efficiency.

[0024] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0025] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0026] Figure 1 A flowchart of a data processing method provided in this embodiment is shown;

[0027] Figure 2 A classification diagram of a decision tree algorithm provided in this embodiment is shown;

[0028] Figure 3 This embodiment illustrates an example of data to be processed.

[0029] Figure 4 This illustration shows a schematic diagram of a column-oriented storage file provided in this embodiment;

[0030] Figure 5 A flowchart of another data processing method provided in this embodiment is shown;

[0031] Figure 6 A flowchart of yet another data processing method provided in this embodiment is shown;

[0032] Figure 7 A flowchart of yet another data processing method provided in this embodiment is shown;

[0033] Figure 8 A flowchart of yet another data processing method provided in this embodiment is shown;

[0034] Figure 9 A flowchart of yet another data processing method provided in this embodiment is shown;

[0035] Figure 10 A flowchart of yet another data processing method provided in this embodiment is shown;

[0036] Figure 11 A flowchart of yet another data processing method provided in this embodiment is shown;

[0037] Figure 12 This illustration shows a schematic diagram of an index file provided in this embodiment;

[0038] Figure 13This diagram illustrates the structure of a data processing apparatus provided in this embodiment.

[0039] Figure 14 A schematic diagram of the structure of an electronic device provided in this embodiment is shown. Detailed Implementation

[0040] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0041] To facilitate the analysis of data generated during business operations, a zipper table can be used to store historical data and record its historical changes. Specifically, a zipper table uses start and end dates to identify whether a record is valid between those dates. Therefore, a zipper table can record all changes from the beginning of a transaction to its current state. However, querying data stored in a zipper table requires careful attention to the start and end dates, making the query logic more complex and resulting in lower data utilization efficiency.

[0042] In view of one or more of the aforementioned issues, Figure 1 The diagram illustrates a data processing method according to an embodiment of the present invention. This method, which can be executed by an electronic device, acquires data to be processed, analyzes the data, determines the target row group of the data to be processed, and writes the data to be processed into the target row group of a columnstore file. This method saves data storage space and improves data utilization efficiency. The electronic device can be a backend server or service cluster of a service provider. Figure 1 As shown, the method may include the following steps:

[0043] Step 110: Obtain the data to be processed.

[0044] The data to be processed may include at least one row of data to be written. The data to be written refers to a single row of data or a single record within the data to be processed.

[0045] During the business process, electronic devices can monitor business changes and automatically generate pending data including one or more rows of data to be written. They can also receive storage requests from users or devices and obtain the pending data carried in the storage request, or they can respond to the storage request and actively obtain the pending data sent by users or devices.

[0046] Step 120: Classify the data to be processed to obtain the classification information of the data to be processed.

[0047] The classification information of the data to be processed can be used to represent the category to which the data belongs. Data to be processed belonging to the same category are more similar. For multiple rows of data, multiple rows of data belonging to the same category are similar in the same attribute.

[0048] To determine the distribution of the data to be processed, the data can be classified into one or more categories.

[0049] For example, a decision tree algorithm can be used to classify the data to be processed and determine its classification information. A decision tree is a tree-like structure where each internal node represents a test on the tree, each branch represents a test output, and each leaf node represents a category. In this embodiment, the decision tree algorithm can be Iterative Dichotomiser 3 (ID3), C4.5, or Classification and Regression Tree (CART) algorithms, etc., and this embodiment does not impose any special limitations on it.

[0050] For example, Figure 2 This embodiment illustrates a classification diagram of a decision tree algorithm. Figure 2 As shown, when using the decision tree algorithm to classify the data to be processed, we can start from the root node and test an attribute feature of the data to be processed, such as attribute feature x. Based on the test results, the data to be processed is assigned to child nodes, such as first-level child node 1, first-level child node 2, ..., first-level child node n. At this time, each child node corresponds to a value of attribute feature x.

[0051] Then, another attribute feature of the data to be processed, such as attribute feature y, can be tested. Based on the test results, the data to be processed is assigned to child nodes, such as second-level child node 1, second-level child node 2, ..., second-level child node n. At this time, each child node corresponds to a value of attribute feature y. All attribute features of the data to be processed are tested and assigned in a recursive manner until a leaf node is reached, at which point the data to be processed is assigned to the class of the leaf node.

[0052] In one alternative approach, a pre-trained decision tree algorithm can be used to classify the data to be processed, obtaining classification information for the data. The pre-trained decision tree algorithm can be generated using acquired historical data. This historical data can be data acquired at a historical time that originates from the same source as the data to be processed.

[0053] Taking the ID3 algorithm as an example, the acquired historical data can be divided into training and test sets. An attribute is selected based on its information entropy, and the training set is then divided according to this attribute. The attribute that results in the largest decrease in the sum of the information entropies of the sub-training sets compared to the undivided dataset is considered the current optimal attribute. After selecting the optimal attribute, the original training set can be further divided into subsets based on its value. Each subset is then treated as a complete dataset, and the selection of the optimal attribute is iteratively repeated until all samples in the dataset share the same classification label, at which point the decision tree generation process ends.

[0054] In addition, to prevent overfitting, the trained decision tree algorithm can be pruned according to the maximum number of nodes in the decision tree algorithm to generate a pre-trained decision tree algorithm.

[0055] By using the above method of classifying the data to be processed, the data can be divided into one or more categories, the distribution of the data can be determined, and the data can be stored in a more convenient way.

[0056] Step 130: Based on the classification information of the data to be processed, determine the target row group corresponding to the classification information among the multiple row groups included in the column storage file.

[0057] Within these multiple row groups, different row groups correspond to different classification information. In other words, the same row group has the same classification information, and the data to be processed within the same row group has a higher similarity. The target row group is the row group used to store the data to be processed.

[0058] Column-oriented storage files are a type of storage file that stores data in a column-oriented storage manner. Figure 3 This embodiment illustrates an example of data to be processed, such as... Figure 3 As shown, the data to be processed consists of 6 rows and 5 columns, where the data in each column is "Serial Number", "Customer Name", "Age", "Location", and "Customer Level". Columnar storage is a storage method that organizes the data in each column together in a certain order. For example, the data in each column can be organized from left to right, such as the data in the "Age" column being 21, 32, 31, 41, 35, and 50.

[0059] In columnar storage, since the data of each column is stored in clusters, when querying data for a few attributes, such as "age" and "customer level", only the data of the corresponding columns, namely the "age" column and the "customer level" column, needs to be read. This can greatly reduce the amount of data read. Furthermore, since the data type corresponding to each attribute is the same, columnar storage can be used to design targeted data compression algorithms.

[0060] In this embodiment, the column-oriented storage file can be a Parquet file. Parquet is a new column-oriented storage format in the Hadoop ecosystem that supports nested data models.

[0061] Figure 4 This illustration shows a schematic diagram of a column-oriented storage file provided in this embodiment, such as... Figure 4 As shown, each end of a Parquet file contains a 4-byte Magic Number with the content "PAR1", used to identify the file as a Parquet file. The data block portion includes multiple line groups, such as... Figure 4 The diagram shows row group 0 and row group 1, each used to store at least one row of data. For example, suppose there are 1000 rows of data, divided into two row groups of appropriate size, with 500 rows in each group. Each row group can include multiple column blocks; for example, row group 0 can include column blocks such as Column a, and row group 1 can include column blocks such as Column b. A column block is composed of a collection of data from a single column. Within each column block, data is stored in pages as the smallest unit.

[0062] Each column block in a Parquet file also requires Repetition Levels, Definition Levels, and Values ​​to store complete information. Values ​​refers to the data values ​​stored in the column block.

[0063] Specifically, Repetition Levels are primarily used to express the length of array-type fields. They indirectly express length by recording changes in the nesting level; if the nesting level remains unchanged, the array continues; if the nesting level changes, the previous array has ended. If the nesting level of a value increases from 0 to 1, then the Repetition level of that value is 0. If the nesting level of a value remains unchanged at a given position, then the Repetition level of that value is its nesting level. For example, for [["a","b"], ["c","d","e"]], its corresponding Repetition level will be encoded as the values ​​shown in Table 1 below:

[0064] Table 1

[0065]

[0066]

[0067] Because the nesting level of this array is 2, and "a" is the boundary from level 0 to level 2, its repetition level is 0. "c" is the boundary from level 1 to level 2, so its repetition level is 1. The nesting level of the other letters has not changed, so their repetition level is 2.

[0068] Definition Levels represent the depth of the definition and are primarily used to indicate the location of null values. Since Parquet files do not explicitly store null values, the definition level is used to determine whether a value is null.

[0069] After obtaining the classification information of the data to be processed, by searching for the target row group that matches the classification information of the data to be processed among the multiple row groups included in the column storage file, it is possible to allocate row groups based on the data characteristics of the data to be processed, which facilitates the subsequent storage of the data to be processed according to the characteristics of the stored data in each row group.

[0070] In an alternative approach, when determining the target row group, the row group with the least amount of data can be selected based on the data volume of each row group in the columnstore file. This ensures a balance in the amount of data stored in each row group.

[0071] Step 140: Write the data to be processed into the target row group in the columnstore file.

[0072] After determining the target row group corresponding to the classification information of the data to be processed, the data to be processed can be written into the target row group in the column storage file. This allows data of the same category to be stored in the same row group as much as possible, which can improve the data compression effect, save storage space, reduce storage costs, and reduce the time spent reading stored data.

[0073] For example, when determining the target row group as such Figure 4 When row group 0 is shown, data to be processed can be written into row group 0 of the column storage file, so that each column of data to be processed is stored in the corresponding column block in row group 0.

[0074] According to the data processing method provided in this embodiment, data to be processed can be obtained, the data to be processed can be classified to obtain classification information of the data to be processed, and then, based on the classification information of the data to be processed, the target row group corresponding to the classification information can be determined among the multiple row groups included in the column storage file, and the data to be processed can be written into the target row group in the column storage file.

[0075] This method allows data to be written to a column-oriented storage file, where the data is stored in target rows with similar characteristics, improving the data compression ratio and saving storage space. At the same time, due to the advantages of the column-oriented storage file format, when reading the stored data, only the data in the corresponding column needs to be read, without having to read every row, thus improving data reading efficiency.

[0076] Considering that in actual business processes, business data may remain unchanged for a period of time, for example, a user may watch the same video content multiple times on the same day. If a record of the name of the content watched by the user is generated each time, it will result in multiple duplicate data entries in the data generated on that day, affecting subsequent business analysis.

[0077] Therefore, in order to reduce duplicate data in stored data, Figure 5 A flowchart of another data processing method provided in this embodiment is shown, such as... Figure 5 As shown, the steps 510-570 may be included:

[0078] Step 510: Obtain the data to be processed.

[0079] The data to be processed may include at least one row of data to be written.

[0080] Step 520: Determine the encoding information of the data to be written in each data row of the data to be processed.

[0081] An encoding algorithm can be an algorithm that converts the data in each row to be written in the data to be processed into data that can uniquely identify the corresponding row to be written. For example, it can be any hash algorithm, such as the Message-Digest Algorithm (MD) and the Secure Hash Algorithm (SHA). The Message-Digest Algorithm, depending on its version, can include MD2, MD4, and MD5. The Secure Hash Algorithm can include the first-generation SHA algorithm standard and the second-generation SHA algorithm, namely SHA-1 and SHA-2. SHA-2 can include algorithms such as SHA-224, SHA-256, SHA-384, and SHA-512.

[0082] Encoded information refers to the encoded data of the data to be written, generated using an encoding algorithm, which can be used to uniquely identify the data to be written.

[0083] For example, any hash algorithm can be used to convert the data of each data row to be written in the data to be processed into a binary string of fixed length, which is the encoding information of the corresponding data row to be written.

[0084] Step 530: Based on the data generation time of the data row to be written, determine the encoding information of at least one stored data row with the same data generation time as the data row to be written in the index file.

[0085] The index file can include the encoding information of multiple stored data rows from the columnstore file. The data generation time of the data row to be written refers to the time when the data for that row was generated; its precision can be selected according to the time dimension of the stored data. The time dimension of the stored data refers to the time dimension in which the data to be processed is stored, which can be one day, one week, or one month, etc.

[0086] For example, for business data that changes frequently, the time dimension for storing the data can be set to a shorter time, while for business data that changes infrequently, the time dimension for storing the data can be set to a longer time.

[0087] Taking a storage time dimension of 1 day as an example, if the data generation time of the data row to be written is January 1, 2023, then the stored data rows of January 1, 2023 can be found in the index file, and the encoding information of the data in these stored data rows can be obtained.

[0088] Step 540: If the encoding information of the data to be written to the data row is different from the encoding information of the data in the already stored data row, then proceed to step 550.

[0089] If the encoding information of the data to be written in the data to be processed is different from the encoding information of the data in the stored data row, it means that there is no identical data in the data to be written in the data row in the time dimension of data storage. Therefore, it can be determined that the data in the data row to be written is not duplicate data, and step 650 can be continued.

[0090] Step 550: Classify the data to be processed to obtain the classification information of the data to be processed.

[0091] In this step, the data to be processed can be the remaining data after the data to be written, after which duplicate data has been discarded (i.e., the data whose encoding information is the same as the data of the data in the stored data row), or it can be the data of all the data rows to be written without discarding duplicate data.

[0092] Step 560: Based on the classification information of the data to be processed, determine the target row group corresponding to the classification information among the multiple row groups included in the column storage file.

[0093] Among these, different row groups correspond to different classification information.

[0094] Based on the classification information of the data to be processed, the target row group corresponding to the classification information can be determined from the multiple row groups included in the column storage file.

[0095] Step 570: Write the data to be processed into the target row group in the columnstore file.

[0096] After determining the target row group, the data to be processed can be written into the target row group in the columnstore file, such as the remaining data after discarding duplicate data.

[0097] Through the above steps 510-570, the data rows to be written that have different encoding information from the data of the stored data rows can be identified. The duplication check of all data rows to be written in the data to be processed is completed. Based on the check results, the data to be processed is classified, the target row group corresponding to the classification information is determined, and the data to be processed is written into the target row group of the column storage file, thereby reducing duplicate data in the column storage file and reducing the storage space occupied.

[0098] It should be noted that the specific implementation methods of steps 510 and 550-570 above can be found in the specific implementation methods of steps 110-140 in the aforementioned embodiments, and will not be repeated here.

[0099] Figure 6 A flowchart of another data processing method provided in this embodiment is shown, such as... Figure 6 As shown, the following steps may be included:

[0100] Step 610: Obtain the data to be processed.

[0101] The data to be processed may include at least one row of data to be written.

[0102] Step 620: Determine the encoding information of the data to be written in each data row of the data to be processed.

[0103] Step 630: Based on the data generation time of the data row to be written, determine the encoding information of at least one stored data row with the same data generation time as the data row to be written in the index file.

[0104] Step 640: If the encoding information of the data to be written is the same as the encoding information of the data in the already stored data row, then discard the data in the data row to be written.

[0105] If, in the data to be processed, the encoding information of a data row to be written is the same as that of a data row that has already been stored, it means that the data row to be written already contains the same data in the time dimension of the corresponding stored data. Therefore, the data row to be written can be determined to be duplicate data, and the data row to be written can be discarded instead of being stored.

[0106] This method allows for duplicate data validation based on the time dimension of the stored data, such as all data for January 1, 2023, as mentioned above, and the deletion of duplicate data in the data to be processed corresponding to that time dimension. Therefore, it reduces the storage space occupied by duplicate data and saves computing resources.

[0107] Step 650: Classify the data to be processed to obtain the classification information of the data to be processed.

[0108] In this step, the data to be processed can be the remaining data after discarding the data of the aforementioned data row to be written in step 640.

[0109] For example, a decision tree algorithm, such as the C4.5 algorithm, can be used to classify the data to be processed in order to obtain the classification information of the data.

[0110] Step 660: Based on the classification information of the data to be processed, determine the target row group corresponding to the classification information among the multiple row groups included in the column storage file.

[0111] Among these, different row groups correspond to different classification information.

[0112] Based on the classification information of the data to be processed, the target row group corresponding to the classification information of the data to be processed can be determined from multiple row groups in the column storage file.

[0113] Step 670: Write the data to be processed into the target row group in the columnstore file.

[0114] After determining the target row group, the data to be processed can be written into the target row group in the columnstore file to complete the storage of the data to be processed.

[0115] Using the above method, we can identify the data rows to be written whose encoded information is different from that of the data rows already stored in the data to be processed. This completes the duplication check of all the data rows to be written in the data to be processed. Based on the check results, we classify the data to be processed, determine the target row group corresponding to the classification information, and write the data to be processed into the target row group of the column storage file. This reduces duplicate data in the column storage file and reduces the storage space occupied.

[0116] In addition, the specific implementation methods of the above steps 610-630 and 650-670 can be found in the specific implementation methods of steps 510-530 and 550-570 in the aforementioned embodiments, and will not be repeated here.

[0117] To improve the efficiency of data processing Figure 7 A flowchart of another data processing method provided in this embodiment is shown, such as... Figure 7 As shown, the following steps may be included:

[0118] Step 710: Obtain the data to be processed.

[0119] The data to be processed may include at least one row of data to be written.

[0120] Step 720: Compare the number of rows of data to be written in the data to be processed with the preset number.

[0121] The number of rows to be written can be the number of rows to be written. The preset number can be customized according to actual needs, such as 100 rows, 500 rows, etc.

[0122] Step 730: If the number of rows of data to be written included in the data to be processed is greater than or equal to the preset number, then proceed to step 740.

[0123] If the number of rows of data to be written included in the data to be processed is greater than or equal to the preset number, it means that the amount of data to be processed has reached the threshold for the amount of data to be classified and processed, so step 740 can be executed.

[0124] Step 740: Classify the data to be processed to obtain the classification information of the data to be processed.

[0125] Step 750: Based on the classification information of the data to be processed, determine the target row group corresponding to the classification information among the multiple row groups included in the column storage file.

[0126] Among these, different row groups correspond to different classification information.

[0127] Step 760: Write the data to be processed into the target row group in the columnstore file.

[0128] After determining the target row group, the data to be processed can be written into the target row group in the columnstore file.

[0129] Using the above method, when the number of rows to be written in the data to be processed reaches a preset number, the data to be processed can be classified, the target row group can be determined, and the data to be processed can be written to the target row group in the column storage file. This realizes a method to determine whether reclassification is needed based on the amount of data to be processed. Moreover, this method does not require classification processing for each row of data, so it can save computing resources and improve data processing efficiency.

[0130] It should be noted that the specific implementation methods of steps 710 and 740-760 above can be found in the specific implementation methods of steps 110-140 in the aforementioned embodiments, and will not be repeated here.

[0131] Figure 8 A flowchart of another data processing method provided in this embodiment is shown, such as... Figure 8 As shown, the following steps may be included:

[0132] Step 810: Obtain the data to be processed.

[0133] The data to be processed may include at least one row of data to be written.

[0134] Step 820: Classify the data of the preset number of data rows to be written to obtain the classification information of the preset number of data rows to be written.

[0135] The preset number of rows to be written is determined based on the generation time of each row in the data to be processed. For example, when determining the preset number of rows to be written, the data of each row can be arranged in order of generation time from earliest to latest, thereby selecting the first N rows that meet the preset number, where the preset number is N.

[0136] When selecting a preset number of rows of data to be written from the data to be processed, these rows can be categorized to determine their classification information. For example, if the data to be processed includes 1000 rows of data to be written, and the preset number is 800, the first 800 rows can be categorized according to their generation time to obtain their classification information.

[0137] Step 830: Based on the classification information of the data of the preset number of data rows to be written, determine the target row group corresponding to the classification information of the preset number of data rows to be written among the multiple row groups included in the column storage file.

[0138] Among these, different row groups correspond to different classification information.

[0139] For example, for the above-mentioned unprocessed data including 1000 rows of data to be written, the target row group corresponding to the classification information of the first 800 rows of data to be written can be determined based on the classification information of the first 800 rows of data to be written.

[0140] Step 840: Write the data of the preset number of data rows to be written into the target row group in the column storage file.

[0141] After determining the target row group, a preset number of rows of data to be written can be written to the target row group in the columnstore file. For example, the data of the first 800 rows of data to be written can be written to the target row group according to the classification information of the first 800 rows of data to be written.

[0142] Using the above method, data that has reached a preset number of rows to be written can be stored according to the number of rows of data to be processed, which can ensure the accuracy of the classification of the data to be processed and improve the processing efficiency of the data to be processed.

[0143] It should be noted that the specific implementation of step 810 above can be found in the specific implementation of step 110 in the aforementioned embodiments, and will not be repeated here.

[0144] Figure 9 A flowchart of another data processing method provided in this embodiment is shown, such as... Figure 9 As shown, the following steps may be included:

[0145] Step 910: Obtain the data to be processed.

[0146] The data to be processed may include at least one row of data to be written.

[0147] Step 920: Classify the data of the preset number of data rows to be written to obtain the classification information of the preset number of data rows to be written.

[0148] The number of rows to be written is determined based on the generation time of each row in the data to be processed.

[0149] Step 930: Based on the classification information of the data of the preset number of data rows to be written, determine the target row group corresponding to the classification information of the preset number of data rows to be written among the multiple row groups included in the column storage file.

[0150] Among these, different row groups correspond to different classification information.

[0151] Step 940: Write the data of the preset number of data rows to be written into the target row group in the column storage file.

[0152] Step 950: Obtain new data to be processed.

[0153] The new data to be processed may include data from at least one new row of data to be written.

[0154] In order to store unclassified rows of data to be written, electronic devices can continuously monitor business changes and acquire new data to be processed, or they can continuously receive new storage requests from users or devices and acquire the new data to be processed carried in the new storage requests, or they can proactively acquire new data to be processed sent by users or devices in response to the new storage requests.

[0155] Step 960: If the sum of the number of unclassified data rows to be written in the data to be processed and the number of new data rows to be written in the new data to be processed is greater than or equal to a preset number, then classify the data of the unclassified data rows to be written and the data of the new data rows to be written, such as the data of all the unclassified data rows to be written and some of the new data rows to be written, to obtain new classification information.

[0156] When acquiring new data to be processed, the sum of the number of unclassified data rows to be written in the data to be processed and the number of new data rows to be written in the new data to be processed can be counted. That is, the total number of unclassified data rows to be written and the total number of new data rows to be written. If the total number is greater than or equal to the preset number, the data of the unclassified data rows to be written and the data of the new data rows to be written can be classified to obtain new classification information.

[0157] For example, for the unclassified data to be written and the new data to be written, all data in both types of data can be classified to obtain new classification information, or data that reaches a preset number in both types of data can be classified to obtain new classification information.

[0158] Step 970: Based on the new classification information, determine the target row group corresponding to the new classification information among the multiple row groups included in the column storage file.

[0159] Among these, different row groups correspond to different classification information.

[0160] Step 980: Write the data of the unclassified data rows to be written from the above-mentioned data to be processed and the data of the new data rows to be written from the new data to be processed into the target row group of the column storage file.

[0161] Since the new classification information can be obtained by classifying the data of the unclassified data rows to be written and the data of the new data rows to be written, or by classifying the data of the two types of data to a preset number, the target row group corresponds to the new classification information.

[0162] Therefore, you can write both types of data into the target row group of the column store file, namely, all the data corresponding to the data of the unclassified data rows to be written and the data of the new data rows to be written. Alternatively, you can write a preset number of these two types of data into the target row group of the column store file, such as the data of all the unclassified data rows to be written and some of the data of the new data rows to be written.

[0163] Using the above method, based on the relationship between the number of unclassified data rows to be written in the data to be processed and the preset number, statistics can be performed on the unclassified data rows to be written and the new data rows to be written in the new data to be processed, and the storage of the unclassified data rows to be written and the new data to be processed can be completed based on the statistical results.

[0164] To improve storage efficiency, in one alternative approach, unclassified rows of data to be written in the data to be processed can be stored directly in the target row group corresponding to the classification information of the classified rows of data to be written in the data to be processed, without classifying the data.

[0165] For example, assuming there are multiple target row groups corresponding to the classification information of the classified data rows to be written, namely row group 1 and row group 2, the row group with the closest data generation time of the unclassified data rows to be written can be determined as the target row group of the unclassified data rows to be written, and the data of the unclassified data rows to be written can be stored in the target row group.

[0166] In one alternative approach, taking a Parquet file as an example of a column storage file, the target row group can include multiple column groups, and each column group can include a data column. Figure 10 A flowchart of another data processing method provided in this embodiment is shown, such as... Figure 10 As shown, the following steps may be included:

[0167] Step 1010: Obtain the data to be processed.

[0168] The data to be processed may include at least one row of data to be written.

[0169] Step 1020: Classify the data to be processed to obtain the classification information of the data to be processed.

[0170] Step 1030: Based on the classification information of the data to be processed, determine the target row group corresponding to the classification information among the multiple row groups included in the column storage file.

[0171] Among these, different row groups correspond to different classification information.

[0172] Step 1040: In the target row group of the column storage file, according to the attribute information of each data column in the data to be processed, store the data of the corresponding data column in the target column group corresponding to the attribute information.

[0173] The target column group can be included in multiple column groups. The attribute information of a data column refers to the data characteristics of a column of data, which can be the column name, data type, etc.

[0174] For example, for such Figure 3 The data to be processed shown can be stored in the target column group corresponding to "Customer Name" according to the attribute information of each data column, such as "Customer Name" and "Age". The data in the "Age" column can be stored in the target column group corresponding to "Age".

[0175] Using the method described above, the data to be processed can be stored in corresponding target column groups according to the attribute information of each data column, thus completing the column storage of the data to be processed. Because the data in a column group within a column store file has a similar data structure, column store files can achieve a higher compression ratio during data compression.

[0176] It should be noted that the specific implementation of steps 1010-1030 above can be found in the specific implementation of steps 110-130 in the aforementioned embodiments, and will not be repeated here.

[0177] After storing the data to be processed in a column store file, the stored data in the column store file can be queried. Specifically, to quickly retrieve the data to be queried, the index file in this embodiment may also include multiple row group identifiers and identifiers of each stored data row in the column store file. The row group identifier is character data that uniquely identifies a row group and can consist of numbers, letters, and special characters; the identifier of a stored data row can be the row number, sequence number, or other unique identifier of a stored data row.

[0178] thus, Figure 11 A flowchart of another data processing method provided in this embodiment is shown, such as... Figure 11 As shown, the following steps may be included:

[0179] Step 1110: Obtain the data to be processed.

[0180] The data to be processed may include at least one row of data to be written.

[0181] Step 1120: Classify the data to be processed to obtain the classification information of the data to be processed.

[0182] Step 1130: Based on the classification information of the data to be processed, determine the target row group corresponding to the classification information among the multiple row groups included in the column storage file.

[0183] Step 1140: Write the data to be processed into the target row group in the columnstore file.

[0184] Step 1150: Obtain the query instruction.

[0185] The query instruction may include the data generation time of the data to be queried.

[0186] For example, a user can select or enter the data generation time of the data to be queried through an input box in a data query interface displayed on a terminal device, such as a computer. The terminal device responds to the user's input of the data generation time, generates a query instruction, and sends the query instruction to the electronic device.

[0187] Step 1160: Based on the data generation time of the data to be queried, determine the row group identifier and the identifiers of multiple stored data rows corresponding to the data generation time of the data to be queried in the index file.

[0188] After obtaining the query command, the electronic device can parse the data generation time of the data to be queried.

[0189] Then, the electronic device can look up the row group identifier corresponding to the data generation time of the data to be queried in the index file, as well as the identifier of the stored data row corresponding to that row group identifier, based on the data generation time of the data to be queried.

[0190] In an alternative approach, the identifier of the stored data row corresponding to the data generation time of the data to be queried, and the row group identifier corresponding to the identifier of the stored data row, can be found in the index file based on the data generation time of the data to be queried.

[0191] Figure 12 This embodiment shows a schematic diagram of an index file, such as... Figure 12As shown, taking Day1 as the data generation time of the data to be queried as an example, we can first look up the identifiers of the stored data rows corresponding to Day1 in the index file, namely rows 0-100, 101-200, and 301-500. Then, based on the identifiers of the stored data rows, we can determine the row group identifiers of rows 0-100, 101-200, and 301-500 corresponding to Day1 from the data of the row group identifiers, namely row group identifier 1, row group identifier 2, and row group identifier 4.

[0192] Step 1170: Determine the data to be queried in the columnstore file based on the row group identifier and the identifiers of multiple stored data rows.

[0193] After determining the row group identifier corresponding to the data generation time of the data to be queried and the identifiers of multiple stored data rows, the row group can be determined in the column store file according to the row group identifier, and the data of the data row corresponding to the identifier of the stored data row can be read from the row group.

[0194] For example, row groups in the column storage file can be determined based on row group identifier 1, row group identifier 2, and row group identifier 4. Then, data from rows 0 to 100 can be read from the row group corresponding to row group identifier 1, data from rows 101 to 200 can be read from the row group corresponding to row group identifier 2, and data from rows 301 to 500 can be read from the row group corresponding to row group identifier 4.

[0195] Using the above method, the row group identifier and the identifier of the stored data row can be determined based on the index file, which can improve data query efficiency, increase reading speed, and reduce the waste of computing resources.

[0196] It should be noted that the specific implementation of steps 1110-1140 above can be found in the specific implementation of steps 110-140 in the foregoing embodiments, and will not be repeated here.

[0197] In summary, according to the data processing method provided in this embodiment, the classification information of the data to be processed can be determined, the target row group can be determined based on the classification information, and the data to be processed can be written into the target row group in the columnar storage file to complete the data storage. Since the data in the same row group corresponds to the same category, similar data can be stored in the same row group as much as possible, which can improve the data compression ratio, save data storage space, and due to the advantages of the columnar storage file format, when reading the stored data, only the data in the corresponding column needs to be read, without having to read every row of data, thus improving the data reading efficiency.

[0198] Figure 13 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present invention is shown, as follows: Figure 13As shown, the data processing device 1300 may include: an acquisition module 1310, which can be used to acquire data to be processed, the data to be processed including at least one data row to be written; a classification module 1320, which can be used to classify the data to be processed to obtain classification information of the data to be processed; a determination module 1330, which can be used to determine the target row group corresponding to the classification information in a column storage file according to the classification information of the data to be processed; different row groups in the multiple row groups correspond to different classification information; and a writing module 1340, which can be used to write the data to be processed into the target row group in the column storage file.

[0199] In one alternative approach, before classifying the data to be processed to obtain classification information, the classification module 1320 can also be used to determine the encoding information of the data to be written in each data row of the data to be processed. Based on the data generation time of the data row to be written, the encoding information of at least one stored data row with the same data generation time as the data row to be written is determined in the index file. The index file includes the encoding information of multiple stored data rows in the column storage file. If the encoding information of the data row to be written is different from the encoding information of the stored data rows, the step of classifying the data to be processed to obtain classification information is performed.

[0200] In an alternative embodiment, the classification module 1320 may also be used to discard the data to be written if the encoding information of the data to be written is the same as the encoding information of the data in the already stored data row.

[0201] In an alternative approach, the index file further includes multiple row group identifiers and identifiers of each stored data row in the column storage file. The write module 1340 can also be used to obtain a query instruction, which includes the data generation time of the data to be queried. Based on the data generation time of the data to be queried, the row group identifier and identifiers of multiple stored data rows corresponding to the data generation time of the data to be queried are determined in the index file. Based on the row group identifier and the identifiers of the multiple stored data rows, the data to be queried is determined in the column storage file.

[0202] In one alternative approach, before classifying the data to be processed and obtaining the classification information of the data to be processed, the classification module 1320 can also be used to compare the number of data rows to be written included in the data to be processed with a preset number. If the number of data rows to be written included in the data to be processed is greater than or equal to the preset number, then the step of classifying the data to be processed and obtaining the classification information of the data to be processed is performed.

[0203] In one alternative approach, the classification module 1320 can be used to classify the data of a preset number of data rows to be written, and obtain the classification information of the preset number of data rows to be written; the preset number of data rows to be written is determined based on the data generation time of each data row to be written in the data to be processed.

[0204] In an alternative approach, the classification module 1320 can also be used to acquire new data to be processed; if the sum of the number of unclassified data rows to be written in the data to be processed and the number of new data rows to be written in the new data to be processed is greater than or equal to a preset number, then the data of the unclassified data rows to be written and the data of the new data rows to be written are classified to obtain new classification information.

[0205] In one alternative approach, the target row group includes multiple column groups, each of which includes a data column. The write module 1340 can be used to store the data of the corresponding data column in the target row group of the data to be processed in the target row group of the column storage file, based on the attribute information of each data column in the data to be processed. The multiple column groups include the target column group.

[0206] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation plan. For details of the undisclosed scheme, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0207] Figure 14 The diagram shows a structural schematic of an electronic device provided by an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0208] like Figure 14 As shown, the electronic device may include: a processor 1402, a communications interface 1404, a memory 1406, and a communications bus 1408.

[0209] The processor 1402, communication interface 1404, and memory 1406 communicate with each other via communication bus 1408. Communication interface 1404 is used to communicate with other network elements such as clients or other servers. The processor 1402 executes program 1410, specifically performing the relevant steps described above in the data processing method embodiment.

[0210] Specifically, program 1410 may include program code, which includes computer-executable instructions.

[0211] Processor 1402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0212] Memory 1406 is used to store program 1410. Memory 1406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0213] Specifically, program 1410 can be called by processor 1402 to cause the electronic device to perform the operation steps of the above data processing method.

[0214] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on an electronic device, causes the electronic device to perform the data processing method described in any of the above method embodiments.

[0215] The executable instructions can be used to cause the electronic device to perform the operation steps of the above data processing method.

[0216] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments of this invention are not directed to any particular programming language.

[0217] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, for the sake of brevity and to aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0218] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0219] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A data processing method, characterized by, The method comprises: acquiring to-be-processed data, the to-be-processed data comprising data of at least one to-be-written data row; classifying the to-be-processed data to obtain classification information of the to-be-processed data; determining, according to the classification information of the to-be-processed data, a target row group corresponding to the classification information among a plurality of row groups comprised in a column storage file; different row groups among the plurality of row groups correspond to different classification information; writing the to-be-processed data in the target row group in the column storage file; before the classifying the to-be-processed data to obtain the classification information of the to-be-processed data, the method further comprises: determining encoding information of data of each to-be-written data row in the to-be-processed data; determining, according to a data generation time of the to-be-written data row, encoding information of data of at least one stored data row with the same data generation time as the to-be-written data row in an index file; the index file comprising encoding information of data of a plurality of stored data rows in the column storage file; if the encoding information of the data of the to-be-written data row is different from the encoding information of the data of the stored data row, performing the classifying the to-be-processed data to obtain the classification information of the to-be-processed data; wherein the index file further comprises a plurality of row group identifiers and identifiers of each stored data row in the column storage file, and the method further comprises: acquiring a to-be-queried instruction, the to-be-queried instruction comprising a data generation time of to-be-queried data; determining, according to the data generation time of the to-be-queried data, a row group identifier and a plurality of stored data row identifiers corresponding to the data generation time of the to-be-queried data in the index file; determining the to-be-queried data in the column storage file according to the row group identifier and the plurality of stored data row identifiers.

2. The method of claim 1, wherein, The method further comprises: if the encoding information of the data of the to-be-written data row is the same as the encoding information of the data of the stored data row, discarding the data of the to-be-written data row.

3. The method according to any one of claims 1-2, characterized in that, Before the classifying the to-be-processed data to obtain the classification information of the to-be-processed data, the method further comprises: comparing a number of to-be-written data rows comprised in the to-be-processed data with a preset number; if the number of to-be-written data rows comprised in the to-be-processed data is greater than or equal to the preset number, performing the classifying the to-be-processed data to obtain the classification information of the to-be-processed data.

4. The method of claim 3, wherein, The classifying the to-be-processed data to obtain the classification information of the to-be-processed data comprises: classifying data of the preset number of to-be-written data rows to obtain classification information of the data of the preset number of to-be-written data rows; The preset number of to-be-written data rows are determined according to data generation times of data of each to-be-written data row in the to-be-processed data.

5. The method of claim 4, wherein, The method further comprises: acquiring new to-be-processed data; If the sum of the number of the unclassified to-be-written data rows in the to-be-processed data and the number of new to-be-written data rows in the new to-be-processed data is greater than or equal to the preset number, the data of the unclassified to-be-written data rows and the data of the new to-be-written data rows are classified to obtain new classification information.

6. The method of any one of claims 1-2, wherein, The target row group includes a plurality of column groups, each column group includes one data column, and the writing of the to-be-processed data into the target row group in the column storage file includes: In the target row group in the column storage file, according to the attribute information of each data column in the to-be-processed data, the data of the corresponding data column in the to-be-processed data is stored in the target column group corresponding to the attribute information; and the plurality of column groups include the target column group.

7. A data processing apparatus, characterized by, The device includes: An acquisition module configured to acquire to-be-processed data, the to-be-processed data including data of at least one to-be-written data row; A classification module configured to classify the to-be-processed data to obtain classification information of the to-be-processed data; A determination module configured to determine, according to the classification information of the to-be-processed data, a target row group corresponding to the classification information from a plurality of row groups included in a column storage file; different row groups in the plurality of row groups correspond to different classification information; A writing module configured to write the to-be-processed data into the target row group in the column storage file; Before the classification of the to-be-processed data to obtain the classification information of the to-be-processed data, the classification module is further configured to determine encoding information of data of each to-be-written data row in the to-be-processed data, generate a time of the data of the to-be-written data row, determine, in an index file, encoding information of data of at least one stored data row having the same data generation time as the to-be-written data row, the index file including encoding information of data of a plurality of stored data rows in the column storage file, and if the encoding information of the data of the to-be-written data row is different from the encoding information of the data of the stored data row, perform the classification of the to-be-processed data to obtain the classification information of the to-be-processed data; The writing module is further configured to acquire a to-be-queried instruction, the to-be-queried instruction including a data generation time of to-be-queried data, determine, according to the data generation time of the to-be-queried data, a row group identifier and identifiers of a plurality of stored data rows corresponding to the data generation time of the to-be-queried data in the index file, and determine the to-be-queried data in the column storage file according to the row group identifier and the identifiers of the plurality of stored data rows, the index file further including a plurality of row group identifiers and identifiers of stored data rows in the column storage file.

8. An electronic device, comprising: includes: a processor; a memory configured to store at least one executable instruction; The executable instruction causes the processor to perform the operations of the data processing method of any one of claims 1-6.

Images