Data management method and device based on row and column mixed storage and electronic equipment

Abstract

The invention relates to a data management method and device based on row and column mixed storage, electronic equipment and a storage medium. The method comprises the steps that target data and a row storage table corresponding to the target data are determined; acquiring a row storage attribute corresponding to the target data in a row storage table; establishing a column storage attribute of the target data according to the row storage attribute; and creating a corresponding column storage table based on the column storage attribute of the target data, and synchronizing data information of the column storage table and the row storage table to a data management center to realize data management of the target data. The problem of automatic synchronization of row and column mixed table metadata under a distributed HTAP database and row and column mixed storage architecture is solved; the internal writing process is shielded for the upper layer application, the usability of the HTAP row and column mixed database is improved, the data consistency is ensured, and the mixed scene that TP real-time online transaction operation and AP class analysis complex query exist in the same table at the same time is better met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data management method, device, electronic device and storage medium based on row-column hybrid storage. Background Art

[0002] Traditional TP (Transaction Processing) and AP (Analytical Processing) separate databases, conduct real-time online transactions in TP, and conduct big data analysis in AP by regularly unloading data from the TP database to the big data platform; it cannot support online real-time massive data aggregation and report generation, and increases the load on the TP and AP platforms during the import and export of large amounts of data. Distributed HTAP databases support both row storage and column storage, meeting the performance requirements of businesses for point-to-point and complex queries.

[0003] However, if only column-based tables and row-based tables are provided, efficient point-check operations and complex query operations cannot be performed on the same table at the same time during the business process of database operations. To solve this problem, if the upper-level business maintains two copies of data storage, the implementation is very complicated and the consistency of metadata and records between row storage and column storage cannot be guaranteed. The entire life cycle of the table needs to be maintained in both row storage and column storage, which greatly increases the complexity of business processing logic, resulting in the separation of TP and AP databases, and the inability to provide complete HTAP database features.

[0004] Therefore, there is an urgent need to develop a data management method, device, electronic device and storage medium based on row-column hybrid storage to solve one or more of the above-mentioned problems. Summary of the invention

[0005] In view of this, in order to solve the above technical problems or part of the technical problems, an embodiment of the present invention provides a data management method, device, electronic device and storage medium based on row-column hybrid storage.

[0006] In a first aspect, the present application provides a data management method based on row-column hybrid storage, the method comprising:

[0007] Determine target data and a row storage table corresponding to the target data;

[0008] Obtaining the row storage attribute corresponding to the target data in the row storage table;

[0009] Establishing the column storage attribute of the target data according to the row storage attribute;

[0010] A corresponding column storage table is created based on the column storage attributes of the target data, and data information of the column storage table and the row storage table is synchronized to a data management center to implement data management of the target data.

[0011] In a possible implementation, after creating a corresponding column storage table based on the column storage attribute of the target data, the method further includes:

[0012] Determine whether the column storage table is successfully established;

[0013] In the case that the column storage table creation fails, the state information of the target data is obtained, and the step of establishing the column storage table attributes of the target data is re-executed using the corresponding new syntax according to the state information of the target data.

[0014] In a possible implementation, the method further includes:

[0015] When the column storage table is created successfully, the state information of the target data is updated, and the updated state information of the target data is pushed to all computing nodes, so that the computing nodes can implement different data operations.

[0016] In a possible implementation, after creating a corresponding column storage table based on the column storage attribute of the target data, the method further includes:

[0017] The data information of the row storage table is synchronized to the column storage table.

[0018] In a possible implementation, the method further includes:

[0019] After creating the corresponding column storage table, if the table metadata is modified, determine the modification information of the table metadata;

[0020] Update the row storage attribute corresponding to the target data according to the modification information;

[0021] Based on the update operation of the row storage attribute, the column storage attribute of the target data is correspondingly updated to achieve synchronous update of the column storage table.

[0022] In a possible implementation, the method further includes:

[0023] After the corresponding column-stored table is created, if the table metadata is modified, stop synchronizing the data information of the row-stored table to the column-stored table.

[0024] In a possible implementation, the method further includes:

[0025] After creating the corresponding column storage table, if the table metadata is deleted, determine the deletion information of the table metadata;

[0026] Determine, according to the deletion information, a column storage to be deleted corresponding to the column storage table and a row storage to be deleted corresponding to the row storage table;

[0027] Execute a deletion operation on the to-be-deleted column storage and the to-be-deleted row storage.

[0028] In a second aspect, the present application provides a data management device based on row-column hybrid storage, the device comprising:

[0029] A determination module, used to determine target data and a row storage table corresponding to the target data;

[0030] An acquisition module, used to acquire the row storage attribute corresponding to the target data in the row storage table;

[0031] An establishing module, used for establishing a column storage attribute of the target data according to the established row storage attribute;

[0032] A creation module is used to create a corresponding column storage table based on the column storage attributes of the target data, and synchronize the data information of the column storage table and the row storage table to a data management center to achieve data management of the target data.

[0033] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data management method based on row-column hybrid storage as described in any one of the embodiments of the first aspect are implemented.

[0034] In a fourth aspect, the present application further provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the data management method based on row-column hybrid storage described in any embodiment of the first aspect are implemented.

[0035] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application solves the problem of automatic synchronization of row-column hybrid table metadata under the distributed HTAP database and row-column hybrid storage architecture; the internal writing process is shielded for upper-layer applications, thereby improving the usability of the HTAP row-column hybrid database and ensuring data consistency, and better meeting the mixed scenario of TP real-time online transaction operations and AP-type complex analysis queries on the same table. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0039] Figure 1 A flowchart of a data management method based on row-column hybrid storage provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of another data management method based on row-column hybrid storage provided in an embodiment of the present application;

[0041] Figure 3 A flowchart of another data management method based on row-column hybrid storage provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the asynchronous process of DDL for a mixed row-column table provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the DDL synchronization process for a mixed row-column table provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of the structure of a data management device based on row-column hybrid storage provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0048] In order to solve the problem that row storage and column storage have their own data storage characteristics in the prior art, row storage is more suitable for querying and updating TP data, and can provide strong consistent transaction characteristics. Column storage uses columns as storage units and is more suitable for AP data access, thereby meeting the needs of data analysis, report generation, etc., resulting in one storage form being unable to fully meet different business scenarios, and if the upper-level business maintains two copies of data storage, the implementation is very complicated, and the consistency of metadata and records between row storage and column storage cannot be guaranteed. It is necessary to perform the entire life cycle maintenance of the table in row storage and column storage respectively, which will greatly increase the logical complexity of business processing, thereby leading to the separation of TP and AP databases, and failing to provide complete HTAP database characteristics. The present application provides a data management method, device, electronic device and storage medium based on row-column hybrid storage, a metadata and data synchronization method between row storage and column storage of a row-column hybrid table, shielding the internal writing process for the upper-level application, improving the ease of use of the HTAP row-column hybrid database, and ensuring data consistency, better meeting the mixed scenario of TP real-time online transaction operations and AP-type complex analysis queries on the same table.

[0049] Figure 1 A flow chart of a data management method based on row-column hybrid storage provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method specifically includes:

[0050] S101, determining target data and a row storage table corresponding to the target data;

[0051] The database row table is a basic data storage structure. In this storage method, each row of data in the table is stored continuously on the disk as a whole. When data is inserted into the row table, the database system writes the data to the storage location on the disk in the order of the rows. Each row record contains the values ​​of all columns defined in the table.

[0052] In this embodiment, firstly, target data to be processed and a row storage table storing the target data are determined.

[0053] S102, obtaining the row storage attribute corresponding to the target data in the row storage table;

[0054] In this embodiment, after the row storage table corresponding to the target data is obtained, the corresponding row storage attributes in the row storage table are obtained, and the row storage attributes include but are not limited to a table identifier, table field information, and the like.

[0055] S103, establishing a column storage attribute of the target data according to the row storage attribute;

[0056] In this embodiment, when establishing the column storage attributes of the target data, the attribute information of the row storage table established before can be referred to, so as to ensure the consistency and accuracy of the data during the establishment process.

[0057] S104: Create a corresponding column storage table based on the column storage attributes of the target data, and synchronize data information of the column storage table and the row storage table to a data management center to implement data management of the target data.

[0058] In this embodiment, after obtaining the column storage attributes, a corresponding column storage table is created based on the column storage attributes of the target data, and then the data information of the column storage table and the row storage table is synchronously updated and integrated into the data management center, so that the data management center can efficiently process and analyze the target data, thereby providing users with accurate data support and services.

[0059] The data management method based on row-column hybrid storage provided in the embodiment of the present application designs a table structure to ensure the read and write performance of all statements. The structure integrates two different storage formats, row storage and column storage, which not only improves the usability of the HTAP row-column hybrid database, but also ensures data consistency, and better adapts to the mixed scenario where TP real-time online transaction operations and AP-type complex analytical queries exist on the same table, thereby providing users with more flexible and powerful data processing capabilities.

[0060] In addition, when creating a table, this application adds an identifier to indicate column storage, supporting row storage, row storage + column storage, and column storage. If no identifier is added, row storage is used by default.

[0061] When creating a row storage table, create the table directly in the row storage engine.

[0062] When creating a column-stored table, the table metadata adds column-stored attributes and pushes them to ZK. After the Coordinator Leader monitors the metadata changes on ZK, it creates it in the column-stored engine. After the column-stored table is created, it is pushed to the metadata management center, the table metadata status is updated, and it is pushed to all computing nodes.

[0063] When creating a row-store + column-store table, create the row-store first and then the column-store table. After the row-store table is created successfully, ZK is used to notify the column-store engine to create the column-store. The process of creating the column-store table is the same as the above process. The column-store table attributes are added to the table metadata information.

[0064] The following specifically discusses creating a row-storage + column-storage table, that is, a mixed row-column table.

[0065] In an optional solution of the embodiment of the present invention, after creating a corresponding column storage table based on the column storage attribute of the target data, the method further includes:

[0066] Determine whether the column storage table is successfully established;

[0067] In the case that the column storage table creation fails, the state information of the target data is obtained, and the step of establishing the column storage table attributes of the target data is re-executed using the corresponding new syntax according to the state information of the target data.

[0068] In this embodiment, after the column storage table in the row-column mixed table is established, it is necessary to judge the establishment process of the proposed column storage table to confirm whether it has been successfully completed;

[0069] Specifically, if the column storage table fails to be created, the current state information of the target data is obtained, and then the steps of establishing the column storage table with the properties of the target data are re-executed using a new grammatical structure according to the state information to ensure that the column storage table can be successfully established.

[0070] In an optional solution of the embodiment of the present invention, the method further includes:

[0071] When the column storage table is created successfully, the state information of the target data is updated, and the updated state information of the target data is pushed to all computing nodes, so that the computing nodes can implement different data operations.

[0072] In this embodiment, when the column storage table is created successfully, the status information of the target data will be updated to ensure that all computing nodes can obtain the latest data status. Then, the updated status information will be pushed to all computing nodes, so that each node can perform corresponding data operations according to the latest data status, whether it is TP transaction processing or AP analysis query.

[0073] In an optional solution of the embodiment of the present invention, after creating a corresponding column storage table based on the column storage attribute of the target data, the method further includes:

[0074] The data information of the row storage table is synchronized to the column storage table.

[0075] In this embodiment, the data information in the row storage table is synchronously updated to the column storage table in real time, ensuring that the data in the two database tables remain consistent and synchronous.

[0076] Figure 2 A flow chart of another data management method based on row-column hybrid storage provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method specifically includes:

[0077] S201. After creating a corresponding column storage table, if the table metadata is modified, determine the modification information of the table metadata;

[0078] In this embodiment, after the column storage table is created and synchronized with the row storage table, if the table metadata is modified, that is, the field type of the data is modified, the field is modified, or the partition is modified, the modification information of the table metadata is first determined.

[0079] S202, updating the row storage attribute corresponding to the target data according to the modification information;

[0080] S203: Based on the update operation of the row storage attribute, the column storage attribute of the target data is correspondingly updated to achieve synchronous update of the column storage table.

[0081] In this embodiment, after the modification information is determined, the row storage attributes of the target data are updated according to the modification information, and then the column storage attributes of the target data are correspondingly updated according to the update operation performed on the row storage attributes, thereby ensuring that the column storage table can be updated synchronously.

[0082] The embodiment of the present application provides a data management method based on row-column hybrid storage. When the table metadata is modified, the specific data table changes are identified, and then the row storage table and the column storage table are updated accordingly according to the data table changes. The row storage table with data changes can be quickly located, and the synchronous update of the column storage attributes is triggered by the update of the row storage attributes, thereby ensuring the real-time and consistency of the data.

[0083] In an optional solution of the embodiment of the present invention, the method further includes:

[0084] After the corresponding column-stored table is created, if the table metadata is modified, stop synchronizing the data information of the row-stored table to the column-stored table.

[0085] In this embodiment, after the corresponding column storage table is constructed, if the table metadata is changed or updated, it is necessary to promptly stop synchronizing the data information in the row storage table to the column storage table to avoid data inconsistency.

[0086] Example: adding column storage attributes to a row storage table

[0087] After adding column storage to a row storage table, full + incremental data synchronization is automatically performed. Adding row storage to a column storage table is not supported, but adding column storage to a row storage table is supported. After adding column storage to a row storage table, full + incremental data synchronization is automatically performed.

[0088] The alter command for modifying the characteristics of a single row-stored or column-stored table is executed directly in the row-stored engine or column-stored engine.

[0089] For the modification of the properties of the alter row-column mixed table, it involves the operation of updating metadata in both the row store and the column store. Now complete the alter or rename operation on the row store, and after all data changes before the alter operation have been synchronized to the column store, stop the row and column store data synchronization of the table. Then perform the alter or rename operation on the column store, complete the column store metadata change, and then resume the row and column data synchronization task of the table.

[0090] Figure 3 A flow chart of another data management method based on row-column hybrid storage provided in an embodiment of the present application is as follows: Figure 3 As shown, the method specifically includes:

[0091] S301. After creating a corresponding column storage table, if the table metadata is deleted, determine the deletion information of the table metadata;

[0092] S302, determining, according to the deletion information, a column storage to be deleted corresponding to the column storage table and a row storage to be deleted corresponding to the row storage table;

[0093] S303: Execute a deletion operation on the to-be-deleted column store and the to-be-deleted row store.

[0094] In this embodiment, after the corresponding column storage table is created, if the table metadata is deleted, it is necessary to first determine the specific information of the deleted table metadata, and then based on the determined deletion information, further specify which column storage tables need to be deleted and which row storage tables contain the row storage data that need to be deleted, and finally perform deletion processing on the column storage data and row storage data that need to be deleted.

[0095] The embodiment of the present application provides a data management method based on row-column hybrid storage. When the table metadata is deleted, the data table changes are obtained and the specific deleted data table is identified. Then, the row storage table and the column storage table are updated accordingly according to the data deletion content. The row storage table of the data deletion can be quickly located, and the synchronous update of the column storage attributes is triggered by the update of the row storage attributes, thereby ensuring the real-time and consistency of the data.

[0096] For example, for a mixed row-column table, both the row storage and column storage tables are dropped at the same time, the metadata is updated through ZK, and the table is deleted after the column storage is aware of it; for a single row storage or column storage table, the drop operation is performed directly in the row storage engine or column storage engine, that is, through the above improvements to the create, alter, and drop DDL processes, the metadata of the row table is uniformly managed, and the data is synchronously integrated to ensure the data consistency of the row and column storage.

[0097] Figure 4 The DDL asynchronous process diagram of the mixed row and column table provided in the embodiment of the present application is as follows: Figure 4 As shown, the DDL asynchronous process is used to implement metadata synchronization of the row-column mixed table for the alter and drop operations of the distributed database row-column mixed table;

[0098] Supports DDL operations on different types of tables through a unified interface on the compute node CN. The following three types of tables are supported:

[0099] NSM Table: a separate row storage table (data is stored only in the row storage data node DN);

[0100] DSM Table: a separate column storage table (data is stored only in the column storage data node XEngine);

[0101] PAX Table: a mixed row-column storage table (exists in both row-storage and column-storage data nodes, and the table data is synchronized from the DN node to the XEngine node based on WAL);

[0102] Since the Alter and Drop processes of mixed row and column tables involve data synchronization, the overall execution process is relatively long and the asynchronous DLL method can be used.

[0103] The process includes:

[0104] Step 1 (create a row storage table): The client initiates the alter and drop table operations of the row-column mixed table through the computing node CN, and first executes the NSM DDL on the row storage node. After successful execution, the metadata management module is notified to update the metadata information, and the table status on zk is updated to 1. After this step is successfully executed, the execution result is returned to the client, and the execution status of the subsequent process is marked by the table status. After the row storage table executes the alter statement, continue to execute the dml statement to update the data, and keep the update record. The synchronization tool synchronizes the column storage data based on the update record.

[0105] Step 2 (Synchronization tool processing): The synchronization tool collects the update logs on the row storage, parses the logs, and plays them back to the column storage engine in real time to achieve data synchronization between the row and column storage. The synchronization tool monitors the metadata status of the table on zk. When it recognizes that the table status is 1, it waits for the data on each shard of the distributed database to complete the synchronization of the column storage table to the alter statement, updates the table status on zk to 2, and stops the synchronization of the row and column data of the table.

[0106] Step 3 (create a column storage table): The column storage engine recognizes that the table status is updated to 2 through ZK, and then performs the table metadata update operation on the column storage engine. After the column storage completes the alter operation, the table status on ZK is updated to 3, and the synchronization tool is notified to resume the data synchronization operation of the table. At this time, the metadata of the table in the row storage and column storage are consistent.

[0107] Step 4 (synchronization tool processing): The synchronization tool monitors the metadata status of the table on zk. When it recognizes that the table status is 3, it resumes the data synchronization operation of the table and synchronizes the data update operations on the table in the row store to the column store in real time.

[0108] Through the above processing flow, the alter asynchronous DDL operation of the row-column mixed table is completed, the row-column storage operation process is automatically coordinated within the database, and the upper-level application is completely shielded, thereby improving the usability of the system.

[0109] Figure 4 The metadata synchronization of mixed row and column tables is realized in an asynchronous way. Because it involves the data synchronization of existing tables, there is a lot of intermediate processing. Under the principle of not affecting data consistency, the response is returned to the client first, thereby reducing the impact on online business. However, for the operation of creating a new table, the table data synchronization function is not involved in the middle, so the synchronous method can be used.

[0110] Figure 5 The DDL synchronization process diagram of the mixed row and column table provided in the embodiment of the present application is as follows: Figure 5 As shown, the synchronization process is as follows:

[0111] Step 1 (execute DDL on row storage table): The client initiates a create DDL operation request to the unified computing node CN. CN notifies the row storage to execute the row storage table NSM create table operation. After successful execution, the CN node pushes the metadata information to the metadata management module. The metadata management module stores the metadata, updates the table status to 1 on the zk node, and returns the execution result to the CN node.

[0112] Step 2 (Execute DDL on column-stored tables): After the CN node receives the message that the row-stored table was created successfully, it notifies the column-stored engine to create the column-stored table. The column-stored engine pulls metadata details from the row-stored engine and then executes the column-stored table creation operation. It notifies the metadata node through the gRPC interface and then updates the table on zk to a 3OK status. The creation result is returned to the CN node, and the CN node returns the final result of the DDL execution to the client.

[0113] Step 3 (data synchronization): The data creation module monitors the status of the table on zk. When it recognizes that a new table has been created and the status is ok, it automatically synchronizes the data from the row storage to the column storage of the table in real time.

[0114] Among them, Figures 4 to 5 In the code, PAXDDL is to execute the DDL SQL statement for mixed row and column tables; NSM DDL is to add the DDL SQL statement for row storage; send to PM push metadata (frm, NSMsDSM) to push metadata (frm file, mixed row and column table attributes) to the MDS module save metadata to save metadata; update metadatafragment to update metadata to zk return metadata with to return the metadata version number; ack clent to the corresponding client; push metadata NSMfinish to push metadata to the column storage handle metadata to process metadata; get more metadata to obtain the full metadata information from the row storage return metadata to return metadata; execut DSM DDL to execute the column storage DDL SQL; gRPC:updatePAxmetadata to update the column storage metadata through the gRPC interface save metadata to save metadata; update PAXmetadata item zk to update metadata information pugh metadata to synchronize metadata to slot and end send data to synchronize data.

[0115] Figure 6 A structural diagram of a data management device based on row-column hybrid storage provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device specifically includes:

[0116] A determination module 601 is used to determine target data and a row storage table corresponding to the target data;

[0117] An acquisition module 602 is used to acquire the row storage attribute corresponding to the target data in the row storage table;

[0118] An establishing module 603, configured to establish a column storage attribute of the target data according to the established row storage attribute;

[0119] The creation module 604 is used to create a corresponding column storage table based on the column storage attributes of the target data, and synchronize the data information of the column storage table and the row storage table to the data management center to achieve data management of the target data.

[0120] In one possible implementation, the creation module 604 is also used to determine the target data and the row storage table corresponding to the target data; obtain the row storage attributes corresponding to the target data in the row storage table; establish the column storage attributes of the target data according to the row storage attributes; create a corresponding column storage table based on the column storage attributes of the target data, and synchronize the data information of the column storage table and the row storage table to the data management center to realize data management of the target data.

[0121] In a possible implementation, the creation module 604 is also used to determine whether the column storage table is successfully established; in the event that the column storage table fails to be created, obtain the status information of the target data, and re-execute the steps of establishing the column storage table attributes of the target data using the corresponding new syntax according to the status information of the target data.

[0122] In a possible implementation, the creation module 604 is further configured to update the state information of the target data when the column storage table is successfully created, and push the updated state information of the target data to all computing nodes so that the computing nodes can implement different data operations.

[0123] In a possible implementation manner, the device further includes a creation module 605 (not shown in the figure) which is also used to synchronize the data information of the row storage table to the column storage table.

[0124] In one possible implementation, the device also includes a modification module 606 (not shown in the figure), which is used to, after creating the corresponding column storage table, determine the modification information of the table metadata if the table metadata is modified; update the row storage attributes corresponding to the target data according to the modification information; based on the update operation of the row storage attributes, correspondingly update the column storage attributes of the target data to achieve synchronous update of the column storage table.

[0125] In a possible implementation, the modification module 606 is further configured to, after creating the corresponding column storage table, stop synchronizing the data information of the row storage table to the column storage table when the table metadata is modified.

[0126] In one possible implementation, the device further includes a deletion module 607 (not shown in the figure), which is used to, after creating the corresponding column storage table, determine the deletion information of the table metadata if the table metadata is deleted; determine the column storage to be deleted corresponding to the column storage table and the row storage to be deleted corresponding to the row storage table according to the deletion information; and execute the deletion operation of the column storage to be deleted and the row storage to be deleted.

[0127] The data management device based on row-column hybrid storage provided in this embodiment can be as follows Figure 6 The data management device based on row-column hybrid storage shown in can execute the following steps: Figure 1-5 All steps of data management based on row-column hybrid storage are implemented Figure 1-5 The technical effect of data management based on row-column hybrid storage is shown in the figure. For details, please refer to Figure 1-5 For the sake of brevity, the relevant description is not repeated here.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, an embodiment of the present application provides an electronic device, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704; the memory 703 is used to store computer programs; the processor 701 is used to execute the program stored in the memory 703, and implement the steps of the data management method based on row-column hybrid storage provided by any of the aforementioned method embodiments:

[0130] Determine the target data and the row storage table corresponding to the target data; obtain the row storage attributes corresponding to the target data in the row storage table; establish the column storage attributes of the target data according to the established row storage attributes; create a corresponding column storage table based on the column storage attributes of the target data, and synchronize the data information of the column storage table and the row storage table to the data management center to realize data management of the target data.

[0131] In one possible implementation, target data and a row storage table corresponding to the target data are determined; row storage attributes corresponding to the target data in the row storage table are obtained; column storage attributes of the target data are established according to the row storage attributes; a corresponding column storage table is created based on the column storage attributes of the target data, and data information of the column storage table and the row storage table are synchronized to a data management center to implement data management of the target data.

[0132] In one possible implementation, it is determined whether the column storage table is successfully established; if the column storage table fails to be created, the status information of the target data is obtained, and the step of establishing the column storage table attributes of the target data is re-executed using the corresponding new syntax according to the status information of the target data.

[0133] In a possible implementation, when the column storage table is created successfully, the state information of the target data is updated, and the updated state information of the target data is pushed to all computing nodes, so that the computing nodes implement different data operations.

[0134] In a possible implementation, the data information of the row storage table is synchronized to the column storage table.

[0135] In one possible implementation, after creating the corresponding column storage table, if the table metadata is modified, the modification information of the table metadata is determined; the row storage attributes corresponding to the target data are updated according to the modification information; based on the update operation of the row storage attributes, the column storage attributes of the target data are correspondingly updated to achieve synchronous update of the column storage table.

[0136] In a possible implementation, after the corresponding column-stored table is created, if the table metadata is modified, synchronization of the data information of the row-stored table to the column-stored table is stopped.

[0137] In one possible implementation, after creating the corresponding column storage table, if the table metadata has been deleted, the deletion information of the table metadata is determined; based on the deletion information, the column storage to be deleted corresponding to the column storage table and the row storage to be deleted corresponding to the row storage table are determined; and the deletion operation of the column storage to be deleted and the row storage to be deleted is performed.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0139] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0140] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data management method based on row-column hybrid storage, characterized in that: include: Determine target data and a row storage table corresponding to the target data; Obtaining the row storage attribute corresponding to the target data in the row storage table; Establishing the column storage attribute of the target data according to the row storage attribute; A corresponding column storage table is created based on the column storage attributes of the target data, and data information of the column storage table and the row storage table is synchronized to a data management center to implement data management of the target data.

2. The method according to claim 1, characterized in that After creating a corresponding column storage table based on the column storage attribute of the target data, the method further includes: Determine whether the column storage table is successfully established; In the case that the column storage table creation fails, the state information of the target data is obtained, and the step of establishing the column storage table attributes of the target data is re-executed using the corresponding new syntax according to the state information of the target data.

3. The method according to claim 2, characterized in that The method further comprises: When the column storage table is created successfully, the state information of the target data is updated, and the updated state information of the target data is pushed to all computing nodes, so that the computing nodes can implement different data operations.

4. The method according to claim 1, characterized in that After creating a corresponding column storage table based on the column storage attribute of the target data, the method further includes: The data information of the row storage table is synchronized to the column storage table.

5. The method according to claim 4, characterized in that The method further comprises: After creating the corresponding column storage table, if the table metadata is modified, determine the modification information of the table metadata; Update the row storage attribute corresponding to the target data according to the modification information; Based on the update operation of the row storage attribute, the column storage attribute of the target data is correspondingly updated to achieve synchronous update of the column storage table.

6. The method according to claim 5, characterized in that The method further comprises: After the corresponding column-stored table is created, if the table metadata is modified, stop synchronizing the data information of the row-stored table to the column-stored table.

7. The method according to claim 1, characterized in that The method further comprises: After creating the corresponding column storage table, if the table metadata is deleted, determine the deletion information of the table metadata; Determine, according to the deletion information, the column storage to be deleted corresponding to the column storage table and the row storage to be deleted corresponding to the row storage table; Execute a deletion operation on the to-be-deleted column storage and the to-be-deleted row storage.

8. A data management device based on row-column hybrid storage, characterized in that: A determination module, used to determine target data and a row storage table corresponding to the target data; An acquisition module, used to acquire the row storage attribute corresponding to the target data in the row storage table; An establishing module, used for establishing a column storage attribute of the target data according to the established row storage attribute; A creation module is used to create a corresponding column storage table based on the column storage attributes of the target data, and synchronize the data information of the column storage table and the row storage table to a data management center to achieve data management of the target data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data management method based on row-column hybrid storage according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data management method based on row-column hybrid storage described in any one of claims 1 to 7 are implemented.

Images