Data management method, system and device for distributed database

Through the classification and database division and table processing of distributed databases, the corresponding relationship between data types and sub-databases is established, which solves the performance problems of the database under high transaction volume, and realizes the improvement of data management efficiency and the integrity of data transmission, satisfying the satisfaction of Different data storage requirements.

CN118916427APending Publication Date: 2024-11-08GUANGDONG POWER GRID CO LTD +1

Patent Information

Application Number
CN202411274529.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The read and write performance of existing databases decreases when processing high transaction volumes, and the database sub-table middleware does not support transactions, resulting in increased database structure complexity and inefficient unified management of multiple data types, making it difficult to improve data management efficiency.

Method used

Through the classification and database division and table processing of distributed databases, the correspondence between data types and sub-databases is established, data migration is realized, data transmission integrity and precise storage are ensured, and distributed computing and data transmission service modules are used to perform the same and different situations. Construct data migration between data sources.

Benefits of technology

It improves database data management efficiency, ensures the integrity and consistency of data transmission, meets the storage needs of different data, and realizes accurate storage and query of business data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916427A_ABST
    Figure CN118916427A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of cloud services, and discloses a data management method, system and device for a distributed database. The method comprises the following steps: when service data is received, classifying the service data based on a distributed database to obtain a plurality of storage data with different data types, and storing the storage data in the distributed database; based on a preset database-dividing and table-dividing algorithm, performing database-dividing and table-dividing processing on the distributed database to form a plurality of sub-databases; and according to the data type of each piece of storage data, establishing a corresponding relationship between the storage data and the sub-database, thereby performing data migration on the storage data. According to the method, the data type of the data stored in the database is defined, data migration between isomerous data sources is achieved, the integrity of data transmission is guaranteed, the storage requirements of different data are met, accurate storage or query of service data is achieved, and the data management efficiency of the database is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cloud service technology, and in particular to a data management method, system and device for a distributed database. Background Art

[0002] With the rapid development of computer technology and network technology, data sharing and information exchange have become increasingly frequent. In the process of data sharing and information exchange, databases play a key role. Among them, the database's core function is to back up key data.

[0003] In the current era of single databases, with the continuous increase in system transaction volume, the database read and write performance has seriously declined. At this time, you can use the sharding middleware, such as mycat and shardingjdbc, to implement sharding and alleviate the read and write performance of a single database. However, the sharding middleware does not support transactions. If you want to ensure data consistency, you need to use distributed transaction middleware, which increases the complexity of the database structure. At the same time, current databases mostly use a unified data storage and management method, and the types of network data are diverse. Although unified management of data can achieve data integrity and consistency, the variety of big data also leads to low data management efficiency in the database. Therefore, how to improve the data management efficiency of the database has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a data management method, system and device for a distributed database, which ensure the integrity of data transmission, realize accurate storage or query of business data, and improve the data management efficiency of the database.

[0005] In order to solve the above technical problems, the present invention provides a data management method for a distributed database in a first aspect, comprising:

[0006] When receiving the business data, classifying the business data based on the distributed database to obtain several types of storage data with different data types, and storing the several types of storage data in the distributed database;

[0007] Based on a preset library and table sharding algorithm, the distributed database is sharded to form a plurality of sub-databases;

[0008] According to the data types of each stored data, establishing corresponding relationships between several types of the stored data and several sub-databases;

[0009] According to the corresponding relationship, data migration is performed on several types of stored data.

[0010] Furthermore, based on the preset library and table sharding algorithm, the distributed database is sharded into libraries and tables to form several sub-databases, specifically:

[0011] Generate splitting rules based on the preset database and table splitting algorithm;

[0012] According to the splitting rule, a splitting function is used to horizontally split the stored data in the distributed database to form a plurality of sub-databases.

[0013] Furthermore, the data migration of several types of the stored data is performed according to the corresponding relationship, specifically:

[0014] When the distributed database is in operation, extracting the stored data in the distributed database;

[0015] Migrating the stored data to a corresponding sub-database according to the corresponding relationship;

[0016] When migrating the stored data, obtaining updated data of the distributed database in real time, parsing and formatting the updated data, and forming incremental data to be stored in the server corresponding to the distributed database;

[0017] When the storage data migration is completed, the incremental data is retrieved from the server corresponding to the distributed database, and the retrieved incremental data is copied to the sub-database.

[0018] Furthermore, the distributed database includes a computing layer, a metadata layer and a storage layer;

[0019] When the computing layer receives the data processing signal, it obtains all cluster information in the metadata layer according to the data processing signal;

[0020] Performing corresponding data processing at a computing layer according to the data processing signal and the cluster information;

[0021] The processed data is stored in a storage layer; wherein the computing layer and the storage layer are arranged on the same server.

[0022] The first aspect of the present invention provides a data management method for a distributed database. When receiving business data, the business data is classified based on the distributed database to obtain several types of storage data with different data types, and the several types of storage data are stored in the distributed database; based on the preset sub-library and sub-table algorithm, the distributed database is sub-library and sub-table processed to form several sub-databases; according to the data type of each storage data, a corresponding relationship is established between the several types of storage data and the several sub-databases; according to the corresponding relationship, data migration is performed on the several types of storage data. The present invention realizes data migration between homogeneous and heterogeneous data sources by clarifying the data type of the data stored in the database, ensures the integrity of data transmission, meets the storage requirements of different data, realizes accurate storage or query of business data, and improves the data management efficiency of the database.

[0023] A second aspect of the present invention provides a data management system for a distributed database, which is used to execute a data management method for a distributed database, including: a distributed database, a data transmission service module, a memory database, a database management module, a database backup module and a database autonomy module;

[0024] The distributed database is used to classify and disperse the business data; the distributed database includes a transactional database and an analytical database;

[0025] The data transmission service module is used for data migration between heterogeneous data sources; wherein the data sources include relational databases, non-relational databases and big data;

[0026] The memory database is used to cache data and provide access to data based on a dual-machine hot standby architecture and a cluster architecture;

[0027] The database management module is used for global data asset management, data governance, database design and development, data integration, data development and data consumption;

[0028] The database backup module is used to perform data backup and data recovery on the database;

[0029] The database autonomy module is used to perform database self-perception, self-repair, self-optimization, self-operation and maintenance, and self-security based on machine learning and expert experience.

[0030] Furthermore, the transactional database further includes:

[0031] Differentiate the levels of each data in the distributed database, and generate level labels to be stored in each data; wherein the level labels include instance-level data labels, database-level data labels, and storage-level data labels;

[0032] When a monitoring signal is received, the data level of the monitoring signal is parsed, a corresponding first-level label is determined according to the data level, and a monitoring data result is extracted from the distributed database according to the first-level label.

[0033] Furthermore, the determining the corresponding first-level label according to the data level further includes:

[0034] The data levels include instance level, database level and storage level;

[0035] The level label corresponding to each data level includes any one or more combinations of instance-level data labels, database-level data labels, and storage-level data labels.

[0036] Furthermore, the analytical database further includes:

[0037] Adopting a distributed computing architecture, the data in the distributed database is computed in shards and stored on the nodes of the analytical database;

[0038] When a query signal is received, a search field of the query signal is parsed, a database node is determined according to the search field, and query data results are extracted on the database node.

[0039] Furthermore, the data transmission service module further includes:

[0040] The data transmission service module includes data filtering, data aggregation, data connection, data sorting, data interception, data merging and conversion components of calculation fields;

[0041] After the data processing is completed using the conversion component of the data transmission service module, the processed data is written into the target data storage repository in real time.

[0042] The second aspect of the present invention provides a data management system for a distributed database, including a distributed database for classifying and distributing business data; a data transmission service module for data migration between heterogeneous data sources, wherein the data sources include relational databases, non-relational databases, and big data. The present invention implements the classified management and distributed storage of business data through a distributed database, making the data type of the data stored in the database more specific and clear; through the data transmission service module, data migration between heterogeneous data sources is implemented, which can ensure the integrity of data transmission and meet the storage requirements of different data, so that users can clearly identify the corresponding database according to their needs, thereby accurately storing or querying business data, thereby improving the data management efficiency of the database.

[0043] A third aspect of the present invention provides a data management device for a distributed database, comprising: a classification module, a database sub-module, a relationship establishment module and a migration module;

[0044] The classification module is used for classifying the business data based on the distributed database when receiving the business data, obtaining several types of storage data with different data types, and storing the several types of storage data in the distributed database;

[0045] The sub-database module is used to perform sub-database and table processing on the distributed database based on a preset sub-database and table algorithm to form a plurality of sub-databases;

[0046] The relationship establishment module is used to establish corresponding relationships between several types of stored data and several sub-databases according to the data types of each stored data;

[0047] The migration module is used to migrate the plurality of types of stored data according to the corresponding relationship.

[0048] The third aspect of the present invention provides a data management device for a distributed database, which is based on the organic combination of modules. When receiving business data, the business data is classified based on the distributed database to obtain several types of storage data with different data types, and the several types of storage data are stored in the distributed database; based on the preset library and table sharding algorithm, the distributed database is sharded to form several sub-databases; according to the data type of each storage data, a corresponding relationship is established between several types of storage data and several sub-databases; according to the corresponding relationship, data migration is performed on several types of storage data. The present invention realizes data migration between homogeneous and heterogeneous data sources by clarifying the data type of the data stored in the database, ensures the integrity of data transmission, meets the storage requirements of different data, realizes accurate storage or query of business data, and improves the data management efficiency of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a flow chart of an embodiment of a data management method for a distributed database provided by the present invention;

[0050] Figure 2 A schematic diagram of the structure of an embodiment of a data management system for a distributed database provided by the present invention;

[0051] Figure 3 A schematic diagram of the structure of an embodiment of a data management device for a distributed database provided by the present invention;

[0052] Figure 4 A schematic diagram of the structure of a computer device provided by the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0055] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0056] Example 1

[0057] See also Figure 1 , is a flow chart of an embodiment of a data management method for a distributed database provided by the present invention, the method includes steps 101 to 104, each step is specifically as follows:

[0058] Step 101: when receiving business data, classify the business data based on a distributed database to obtain several types of storage data with different data types, and store the several types of storage data in the distributed database.

[0059] Furthermore, in the first embodiment of the present invention, the business data can be uploaded by the terminal, or acquired by the distributed database through real-time monitoring of the terminal. When the distributed database receives the business data, it performs sharding calculations on the business data, classifies the data according to preset data classification categories, extracts different types of data from the business data, forms several types of storage data, and stores them in a distributed database.

[0060] Step 102: Based on a preset library and table partitioning algorithm, the distributed database is partitioned into libraries and tables to form a plurality of sub-databases.

[0061] Furthermore, in the first embodiment of the present invention, based on a preset library-table partitioning algorithm, the distributed database is partitioned into libraries and tables to form a plurality of sub-databases, specifically:

[0062] Generate splitting rules based on the preset database and table splitting algorithm;

[0063] According to the splitting rule, a splitting function is used to horizontally split the stored data in the distributed database to form a plurality of sub-databases.

[0064] In the first embodiment of the present invention, database sharding is the process of splitting a single database into multiple databases, and scattering data in multiple databases. Table sharding is the process of splitting a single table into multiple tables, and scattering data in multiple tables. Based on the database and table sharding algorithm, a splitting rule can be generated. According to the splitting rule, a suitable splitting function is selected from the splitting functions that fit the business in multiple dimensions, and the business data is horizontally split and the results are merged.

[0065] As an example of the first embodiment of the present invention, when the database storage space is insufficient and the database needs to be expanded, or when the database needs to be sharded according to actual usage requirements, the user selects a split key, and the distributed database can generate split rules according to the split key, perform horizontal splitting of data and complex result merging, and shard the current database. The present invention supports a variety of sharding algorithms. Sharding can effectively alleviate the performance bottleneck and pressure of a single machine and a single database, and break through the bottlenecks of IO, number of connections, hardware resources, etc.

[0066] Step 103: establishing corresponding relationships between several types of stored data and several sub-databases according to the data types of each stored data.

[0067] Step 104: performing data migration on the plurality of types of stored data according to the corresponding relationship.

[0068] Furthermore, in the first embodiment of the present invention, data migration is performed on several types of stored data according to the corresponding relationship, specifically:

[0069] When the distributed database is in operation, extracting the stored data in the distributed database;

[0070] Migrating the stored data to a corresponding sub-database according to the corresponding relationship;

[0071] When migrating the stored data, obtaining updated data of the distributed database in real time, parsing and formatting the updated data, and forming incremental data to be stored in the server corresponding to the distributed database;

[0072] When the storage data migration is completed, the incremental data is retrieved from the server corresponding to the distributed database, and the retrieved incremental data is copied to the sub-database.

[0073] In the first embodiment of the present invention, data migration is the process of selecting, preparing, extracting and converting data and permanently transferring it from one computer storage system to another computer storage system. Data migration methods include full migration and incremental migration. Full data migration means that during the full data migration phase, the data transmission service will migrate all the existing data of the source database to the target database. The source database remains in operation and continues to update data during the migration process. The data transmission service uses an incremental data reading module to obtain data updates that occur during the full data migration process. When the full data migration begins, the incremental data reading module will be activated. During the full data migration phase, the incremental data will be parsed, reformatted and stored on the local data transmission server. Incremental data migration means that when the full data migration is completed, the data transmission service will retrieve the locally stored incremental data, reformat it, and apply the data updates to the target database. This process will continue until all ongoing data changes are copied to the target database and the source database and the target database are fully synchronized.

[0074] Further, in the first embodiment of the present invention, the distributed database includes a computing layer, a metadata layer and a storage layer;

[0075] When the computing layer receives the data processing signal, it obtains all cluster information in the metadata layer according to the data processing signal;

[0076] Performing corresponding data processing at a computing layer according to the data processing signal and the cluster information;

[0077] The processed data is stored in a storage layer; wherein the computing layer and the storage layer are arranged on the same server.

[0078] In the first embodiment of the present invention, a distributed database is a database that stores data in different physical locations. The distributed database itself is divided into a computing layer, a metadata layer, and a storage layer. Among them, the computing layer is the SQL (Structured Query Language) layer, which is used to perform operations such as permission checks and routing access on data access. The metadata layer records how many storage nodes there are under the distributed database cluster, and what the corresponding metadata information such as IP and port is. When the computing layer of the distributed database is started, it will first access the metadata layer to obtain all cluster information in order to correctly perform SQL parsing and routing. In addition, because the metadata information is stored in the metadata layer, the distributed database can have multiple computing layers to achieve performance expansion. The storage layer is used to store data, and the storage layer must be on the same server as the computing layer.

[0079] In summary, the first embodiment of the present invention provides a data management method for a distributed database. When receiving business data, based on the distributed database, the business data is classified to obtain several types of storage data with different data types, and the several types of storage data are stored in the distributed database; based on the preset sub-library and table algorithm, the distributed database is sub-library and table processed to form several sub-databases; according to the data type of each storage data, a corresponding relationship is established between the several types of storage data and the several sub-databases; according to the corresponding relationship, data migration is performed on the several types of storage data. The present invention realizes data migration between homogeneous and heterogeneous data sources by clarifying the data type of the data stored in the database, ensures the integrity of data transmission, meets the storage requirements of different data, realizes accurate storage or query of business data, and improves the data management efficiency of the database.

[0080] Example 2

[0081] See also Figure 2 , is a schematic diagram of the structure of an embodiment of a data management system for a distributed database provided by the present invention, the system is used to execute a data management method for a distributed database, including: a distributed database 201, a data transmission service module 202, a memory database 203, a database management module 204, a database backup module 205 and a database autonomy module 206;

[0082] The distributed database 201 is used to classify and disperse the business data; the distributed database 201 includes a transactional database 2011 and an analytical database 2012 .

[0083] Furthermore, in the second embodiment of the present invention, the transactional database 2011 further includes:

[0084] Differentiate the levels of each data in the distributed database 201, and generate level labels to be stored in each data; wherein the level labels include instance-level data labels, database-level data labels, and storage-level data labels;

[0085] When a monitoring signal is received, the data level of the monitoring signal is parsed, a corresponding first-level label is determined according to the data level, and monitoring data results are extracted from the distributed database 201 according to the first-level label.

[0086] Furthermore, in the second embodiment of the present invention, determining the corresponding first-level label according to the data level also includes:

[0087] The data level includes instance level, database level and storage level;

[0088] The level label corresponding to each data level includes any one or more combinations of instance-level data labels, database-level data labels, and storage-level data labels.

[0089] In the second embodiment of the present invention, the transactional database 2011 is used to provide distributed storage services and to perform distributed storage and management of business data. The transactional database 2011 is the main application of the traditional relational database, representing a system with very high transactionality, which is generally a highly available online system, mainly based on small transactions and small queries. When evaluating its system, the number of business transactions and execute SQLs executed per second is generally considered. The transactional database 2011 is mainly used for basic and daily transaction processing, such as bank transactions. The transactional database 2011 emphasizes database memory efficiency, the command rate of various memory indicators, binding variables, and concurrent operations.

[0090] In the second embodiment of the present invention, the transactional database 2011 includes a hierarchical monitoring unit and a data splitting management unit. The hierarchical monitoring unit is used to provide a data hierarchical monitoring service of at least one data level, so that users can clearly and intuitively view monitoring data of different dimensions. Among them, the data level includes instance level, database level and storage level. The data splitting management unit is used to generate splitting rules based on the library and table splitting algorithm, and based on the splitting rules, horizontally split the business data and merge the results.

[0091] As an example of the second embodiment of the present invention, the data level is an important relationship for expressing data, and the instance level, database level and storage level can be a progressive data hierarchy structure, that is, storage level>database level>instance level. For example, the instance level can be instance monitoring data for individual users or devices, the database level can be database-level monitoring data corresponding to the distributed subsystem, and the storage level can be monitoring data for monitoring the target system as a whole, etc. The data at each level can be distinguished by using label encoding. For example, an exclusive label corresponding to each data level can be specified, and then the exclusive label can be inserted into the data header of each data level to achieve classification of monitoring data in different dimensions. Therefore, when labeling different data levels, labels of multiple levels can also be included. For example, instance-level monitoring data can include instance-level data labels, database-level data labels and storage-level data labels corresponding to the target monitoring data.

[0092] As an example of the second embodiment of the present invention, the transactional database 2011 may be a TP database (Transaction Processing), which takes transaction processing as the core and focuses on data consistency and integrity. It is used in high-concurrency, high-transaction processing scenarios, such as banking systems, securities trading systems, etc. The transactional database 2011 may be compatible with MySQL or Oracle protocols and syntax, support sub-libraries and sub-tables, smooth expansion, service upgrade and downgrade, transparent read-write separation, and distributed transactions, and has the operation and maintenance management capabilities of the distributed database throughout its life cycle.

[0093] Furthermore, in the second embodiment of the present invention, the analytical database 2012 further includes:

[0094] Using a distributed computing architecture, the data in the distributed database 201 is computed in shards and stored on the nodes of the analytical database 201;

[0095] When a query signal is received, a search field of the query signal is parsed, a database node is determined according to the search field, and query data results are extracted on the database node.

[0096] In the second embodiment of the present invention, the analytical database 2012 is the main application of the data warehouse system, supports complex analytical operations, focuses on decision support, and provides intuitive and easy-to-understand query results. The analytical database 2012 emphasizes data analysis, SQL execution market, disk I / O, partitioning, etc. The analytical database 2012 is used for high-concurrency writing and updating of PB-level data, and provides interactive analysis services to realize data query of big data. Among them, the interactive analysis service can give users the ability to perform multi-dimensional and interactive statistical analysis on massive data, so as to make full use of the value of data for quantitative operations, auxiliary decision-making, etc. Interactive analysis services are mainly used in report statistics, ad hoc queries and other fields. The former query mode is relatively fixed, for example, typical real-time reports such as PV (Page View, page views), UV (User View, independent user volume), and activity in the mobile Internet; the latter improvises exploratory analysis, such as ad hoc queries such as crowd insights and correlation analysis in the field of Internet content.

[0097] In the second embodiment of the present invention, the analytical database 2012 generally includes multiple control nodes and multiple computing nodes. The control nodes are responsible for compiling computing tasks, generating execution plans, and aggregating computing results, while the computing nodes are responsible for computing tasks divided into specific database instances.

[0098] As an example of the second embodiment of the present invention, the analytical database 2012 can adopt a distributed computing architecture and use data sharding to achieve parallel computing. For example, the Hash mode can be used for data sharding, which is suitable for fact tables or large tables. The data is distributed to a certain node according to the hash value of a certain field or a combination of fields in a record. The hash function can have multiple modes. By distributing data according to the hash value of a given field, a large table can be evenly distributed to multiple nodes of the analytical database 2012. When querying this table, the analytical database compiler can quickly locate the query to one or several related database nodes according to the corresponding search field in the SQL and send the SQL down, and the corresponding database node can quickly respond to the query results.

[0099] As an example of the second embodiment of the present invention, the analytical database 2012 can be an AP database (Availability and Partition Tolerance), which is a database with availability and partition fault tolerance as the core, focusing on the high availability of data and data partitioning in a distributed environment. It is used for distributed data storage in a large-scale network environment, such as social networks, search engines, etc. The analytical database 2012 is used to connect the big data ecosystem, support high concurrency and low latency analysis and processing of PB (petabyte, quadrillion bytes) level data, and can be easily connected with mainstream business intelligence analysis BI (Business Intelligence) tools (such as Tableau, Fanruan, etc.) to carry out BI visualization analysis or ad hoc query. The analytical database 2012 supports real-time data, supports high-concurrency real-time writing and updating, and the writing speed can reach 100 million TPS (Transactions Per Second, the number of transactions transmitted per second), and data can be checked as soon as it is written. It supports complex queries of massive data, fully parallel computing, and achieves sub-second response for PB-level data association analysis. It supports massive data point queries, provides PB-level storage, and writes and queries billions of records per second. It supports seamless integration of federated queries with offline batch computing platforms, without moving data, and allows for direct interactive analysis and quick query results. You can query the table data in the data warehouse directly or combine it with real-time data for joint computing.

[0100] The data transmission service module 202 is used for data migration between heterogeneous data sources; wherein the data sources include relational databases, non-relational databases and big data.

[0101] Furthermore, in the second embodiment of the present invention, the data transmission service module 202 further includes:

[0102] The data transmission service module 202 includes data filtering, data aggregation, data connection, data sorting, data interception, data merging, and conversion components for calculating fields;

[0103] After the data processing is completed using the conversion component of the data transmission service module 202, the processed data is written into the target data storage repository in real time.

[0104] In the second embodiment of the present invention, the data transmission service module 202 is used to realize data migration between different databases, and migrate the data in the current database to the sub-database, so as to store the storage data corresponding to each data type through the sub-database. The data transmission service module 202 includes conversion components such as data filtering, data aggregation, data connection, data sorting, data interception, data merging, and calculation fields, which can realize rich conversion operations and write the processed data into the target database in real time.

[0105] As an example of the second embodiment of the present invention, the data transmission service module 202 supports data sources such as relational databases, NoSQL (Not Only SQL, non-relational databases), big data (OLAP), etc., integrates data migration, subscription and real-time synchronization functions, and solves the problem of long-distance and second-level asynchronous data transmission.

[0106] As an example of the second embodiment of the present invention, the data transmission service module 202 adopts a remote multi-active architecture to provide real-time data flow for downstream applications. It supports streaming ETL (Extract-Transform-Load) of the data synchronization process, adds data input, data output, field selection between the source library and the target library, and the data transmission service module 202 supports data migration between the same and heterogeneous data sources.

[0107] The memory database 203 is used to cache data and provide access to data based on a dual-machine hot standby architecture and a cluster architecture.

[0108] Many websites and applications today typically store their data in a database. Reading and writing data from a database can significantly impact the latency of your application. Writing to a database adds latency because databases typically write data to disk rather than keeping it in memory. It is therefore common for databases to apply compression and encryption, which adds latency when reading and writing data.

[0109] In the second embodiment of the present invention, the memory database 203 uses RAM (Random Access Memory) instead of a hard disk drive (HDD) or a solid state drive (SSD) to store data, which greatly reduces the delay in reading and writing data. First, accessing data from the memory is faster than accessing data from the disk, and secondly, the data structure used to store data in the memory is more direct than disk storage, so the memory database 203 of the present invention can reduce delays when reading and writing data.

[0110] In the second embodiment of the present invention, RAM is an internal memory that directly exchanges data with the CPU, which can be read and written at any time and at a very high speed. Optimizing the data structure used by the memory database 203 can speed up the retrieval speed. For example, relational databases usually use B-trees for indexing, allowing fast searches while supporting reading and writing large data blocks to disk. The memory database 203 of the present invention does not need to write data blocks to disk, and can select a data structure with stronger performance to further reduce latency. The memory database 203 usually stores and uses data as is, without any conversion or parsing at the database layer. The present invention speeds up the time to read and write data, thereby increasing the reduction of latency.

[0111] In the second embodiment of the present invention, the memory database 203 can provide cache database instances of various specifications, support management operations such as instance creation, restart, release, and backup, support clearing all data and clearing expired data in the console, and support network isolation of instances and other functions.

[0112] The database management module 204 is used for global data asset management, data governance, database design and development, data integration, data development and data consumption.

[0113] In the second embodiment of the present invention, the database management module 204 can standardize the research and development, from the offline environment to fully self-service table structure design according to the design specifications, to SQL audit used in the code before going online, to publishing to the designated environment for complete database research and development process support.

[0114] In the second embodiment of the present invention, the database management module 204 can manage and control the fine-grained operation permissions at the library, table, and field levels, and all database operations generated by users through the product can be online and traceable for audit support.

[0115] In the second embodiment of the present invention, the database management module 204 can provide operation specifications and approval process support for various modules such as structural design, data change, data export, and permission application that can be flexibly configured according to business needs, such as approval process operation management such as adding, editing, and deleting approval templates and approval nodes.

[0116] In the second embodiment of the present invention, the database management module 204 can unify the entrance for developing and interacting with the database, so that no user has to directly access the database account password or frequently switch database links for management.

[0117] In the second embodiment of the present invention, the database management module 204 can provide an online task scheduling function to implement database SQL task periodic scheduling and management, and easily solve functions such as historical data dumping and periodic report analysis and output.

[0118] In the second embodiment of the present invention, the database management module 204 can uniformly manage the distribution of sensitive data; support metadata automatic scanning, and can timely discover and effectively manage sensitive data according to custom classification and grading templates; support sensitive data desensitization management, and built-in desensitization algorithms to achieve differentiated desensitization requirements for different scenarios and different fields; support monitoring, abnormal auditing and early warning of sensitive data, and solve the problem of abnormal use of sensitive data and data leakage. Provide official product documents or product function screenshots.

[0119] The database backup module 205 is used to perform data backup and data recovery on the database.

[0120] In the second embodiment of the present invention, the database backup module 205 can provide continuous data protection and low-cost backup services for the database. It can provide strong protection for data in a variety of environments, including enterprise data centers, other cloud vendors and public clouds. The database backup module 205 has a complete set of data backup and data recovery solutions, with real-time incremental backup and data recovery capabilities accurate to seconds.

[0121] The database autonomy module 206 is used to perform database self-perception, self-repair, self-optimization, self-operation and maintenance, and self-security based on machine learning and expert experience.

[0122] In the second embodiment of the present invention, the database autonomy module 206 can provide a cloud service based on machine learning and expert experience to achieve database self-perception, self-repair, self-optimization, self-operation and maintenance, and self-security, helping users eliminate the complexity of database management and service failures caused by manual operations, and effectively ensuring the stability, security, and efficiency of database services.

[0123] In the second embodiment of the present invention, the database autonomous module 206 can provide a visual database performance, resource, and risk monitoring tool; support diagnostic reports, and provide one-click generation of database diagnostic reports, the report content includes performance capacity, slow SQL analysis, space diagnosis, session diagnosis and other contents.

[0124] In the second embodiment of the present invention, the database autonomous module 206 can support the SQL analysis function, execute a SQL task process to view it once, and then perform statistical analysis on all result sets. It can be clearly seen which SQLs are executed the most times during the collection interval, whether there are slow SQLs, etc.

[0125] In the second embodiment of the present invention, the database autonomous module 206 can perform real-time active session analysis and multi-dimensional analysis and statistical analysis functions, and support rapid positioning of users and application servers with the largest number of active sessions.

[0126] In the second embodiment of the present invention, the database autonomous module 206 can automatically collect database performance information at regular intervals through icons or black screens, effectively helping to monitor and escort the database.

[0127] In the second embodiment of the present invention, the database autonomous module 206 can automatically identify suspended transactions, execute abnormal SQL, block SQL, and can directly kill, limit or optimize SQL.

[0128] In summary, the second embodiment of the present invention provides a data management system for a distributed database, including a distributed database for classifying and distributing storage of business data; a data transmission service module for data migration between heterogeneous data sources, wherein the data sources include relational databases, non-relational databases, and big data. The present invention implements the classified management and distributed storage of business data through a distributed database, making the data type of data stored in the database more specific and clear; through the data transmission service module, data migration between heterogeneous data sources is implemented, which can ensure the integrity of data transmission and meet the storage requirements of different data, so that users can clearly identify the corresponding database according to their needs, thereby accurately storing or querying business data, thereby improving the data management efficiency of the database.

[0129] Example 3

[0130] See also Figure 3 , is a schematic diagram of the structure of an embodiment of a data management device for a distributed database provided by the present invention, the device includes a classification module 301, a database sub-module 302, a relationship establishment module 303 and a migration module 304;

[0131] The classification module 301 is used for classifying the business data upon receiving the business data based on the distributed database, obtaining several types of storage data with different data types, and storing the several types of storage data in the distributed database;

[0132] The database partitioning module 302 is used to perform database partitioning and table partitioning processing on the distributed database based on a preset database partitioning and table partitioning algorithm to form a plurality of sub-databases;

[0133] The relationship establishment module 303 is used to establish corresponding relationships between several types of stored data and several sub-databases according to the data types of each stored data;

[0134] The migration module 304 is used to migrate the plurality of types of stored data according to the corresponding relationship.

[0135] Furthermore, in the third embodiment of the present invention, based on the preset library and table sharding algorithm, the distributed database is sharded to form several sub-databases, specifically:

[0136] Generate splitting rules based on the preset database and table splitting algorithm;

[0137] According to the splitting rule, a splitting function is used to horizontally split the stored data in the distributed database to form a plurality of sub-databases.

[0138] Furthermore, in a third embodiment of the present invention, data migration is performed on several types of stored data according to the corresponding relationship, specifically:

[0139] When the distributed database is in operation, extracting the stored data in the distributed database;

[0140] Migrating the stored data to a corresponding sub-database according to the corresponding relationship;

[0141] When migrating the stored data, obtaining updated data of the distributed database in real time, parsing and formatting the updated data, and forming incremental data to be stored in the server corresponding to the distributed database;

[0142] When the storage data migration is completed, the incremental data is retrieved from the server corresponding to the distributed database, and the retrieved incremental data is copied to the sub-database.

[0143] Further, in a third embodiment of the present invention, the distributed database includes a computing layer, a metadata layer, and a storage layer;

[0144] When the computing layer receives the data processing signal, it obtains all cluster information in the metadata layer according to the data processing signal;

[0145] Performing corresponding data processing at a computing layer according to the data processing signal and the cluster information;

[0146] The processed data is stored in a storage layer; wherein the computing layer and the storage layer are arranged on the same server.

[0147] In summary, the third embodiment of the present invention provides a data management device for a distributed database, which is based on the organic combination of modules. When receiving business data, the business data is classified based on the distributed database to obtain several types of storage data with different data types, and the several types of storage data are stored in the distributed database; based on the preset library and table sharding algorithm, the distributed database is sharded to form several sub-databases; according to the data type of each storage data, a corresponding relationship is established between several types of storage data and several sub-databases; according to the corresponding relationship, data migration is performed on several types of storage data. The present invention realizes data migration between homogeneous and heterogeneous data sources by clarifying the data type of the data stored in the database, ensures the integrity of data transmission, meets the storage requirements of different data, realizes accurate storage or query of business data, and improves the data management efficiency of the database.

[0148] The distributed database data management method provided in the above embodiment can be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.

[0149] See also Figure 4 , is a structural diagram of a computer device provided in this statement, the computer device may be a server, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0150] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any data management method of a distributed database.

[0151] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0152] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any data management method of a distributed database.

[0153] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0154] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0155] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any distributed database data management method provided in the embodiment of the present application.

[0156] The computer-readable storage medium may be an internal storage unit of the computer device described in the above embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0157] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data management method for a distributed database, It is characterized in that include: When receiving the business data, classifying the business data based on the distributed database to obtain several types of storage data with different data types, and storing the several types of storage data in the distributed database; Based on a preset library and table sharding algorithm, the distributed database is sharded to form a plurality of sub-databases; According to the data types of each stored data, establishing corresponding relationships between several types of the stored data and several sub-databases; According to the corresponding relationship, data migration is performed on several types of stored data.

2. The data management method of a distributed database according to claim 1, It is characterized in that Based on the preset library and table sharding algorithm, the distributed database is sharded to form several sub-databases, specifically: Generate splitting rules based on the preset database and table splitting algorithm; According to the splitting rule, a splitting function is used to horizontally split the stored data in the distributed database to form a plurality of sub-databases.

3. The data management method of a distributed database according to claim 2, It is characterized in that The data migration of the plurality of types of stored data according to the corresponding relationship is specifically as follows: When the distributed database is in operation, extracting the stored data in the distributed database; Migrating the stored data to a corresponding sub-database according to the corresponding relationship; When migrating the stored data, obtaining updated data of the distributed database in real time, parsing and formatting the updated data, and forming incremental data to be stored in the server corresponding to the distributed database; When the storage data migration is completed, the incremental data is retrieved from the server corresponding to the distributed database, and the retrieved incremental data is copied to the sub-database.

4. The data management method of a distributed database according to claim 1, It is characterized in that The distributed database includes a computing layer, a metadata layer and a storage layer; When the computing layer receives the data processing signal, it obtains all cluster information in the metadata layer according to the data processing signal; Performing corresponding data processing at a computing layer according to the data processing signal and the cluster information; The processed data is stored in a storage layer; wherein the computing layer and the storage layer are arranged on the same server.

5. A data management system for a distributed database, It is characterized in that A method for managing a distributed database according to any one of claims 1 to 4, comprising: a distributed database, a data transmission service module, a memory database, a database management module, a database backup module and a database autonomy module; The distributed database is used to classify and disperse the business data; the distributed database includes a transactional database and an analytical database; The data transmission service module is used for data migration between heterogeneous data sources; wherein the data sources include relational databases, non-relational databases and big data; The memory database is used to cache data and provide access to data based on a dual-machine hot standby architecture and a cluster architecture; The database management module is used for global data asset management, data governance, database design and development, data integration, data development and data consumption; The database backup module is used to perform data backup and data recovery on the database; The database autonomy module is used to perform database self-perception, self-repair, self-optimization, self-operation and maintenance, and self-security based on machine learning and expert experience.

6. The data management system for a distributed database according to claim 5, It is characterized in that The transactional database further includes: Differentiate the levels of each data in the distributed database, and generate level labels to be stored in each data; wherein the level labels include instance-level data labels, database-level data labels, and storage-level data labels; When a monitoring signal is received, the data level of the monitoring signal is parsed, a corresponding first-level label is determined according to the data level, and a monitoring data result is extracted from the distributed database according to the first-level label.

7. The data management system for a distributed database according to claim 6, It is characterized in that The step of determining the corresponding first-level label according to the data level further includes: The data levels include instance level, database level and storage level; The level label corresponding to each data level includes any one or more combinations of instance-level data labels, database-level data labels, and storage-level data labels.

8. The data management system for a distributed database according to claim 5, It is characterized in that The analytical database further includes: Adopting a distributed computing architecture, the data in the distributed database is computed in shards and stored on the nodes of the analytical database; When a query signal is received, a search field of the query signal is parsed, a database node is determined according to the search field, and query data results are extracted on the database node.

9. The data management system for a distributed database according to claim 5, It is characterized in that The data transmission service module further includes: The data transmission service module includes data filtering, data aggregation, data connection, data sorting, data interception, data merging and conversion components of calculation fields; After the data processing is completed using the conversion component of the data transmission service module, the processed data is written into the target data storage repository in real time.

10. A data management device for a distributed database, It is characterized in that include: Classification module, database sub-module, relationship building module and migration module; The classification module is used for classifying the business data based on the distributed database when receiving the business data, obtaining several types of storage data with different data types, and storing the several types of storage data in the distributed database; The sub-database module is used to perform sub-database and table processing on the distributed database based on a preset sub-database and table algorithm to form a plurality of sub-databases; The relationship establishment module is used to establish corresponding relationships between several types of stored data and several sub-databases according to the data types of each stored data; The migration module is used to migrate the plurality of types of stored data according to the corresponding relationship.

Citation Information

Patent Citations

  • Distributed service data access method based on engine

    CN105677915A

  • Distributed data area system

    CN114385578A

  • Data migration method and device, equipment and storage medium

    CN115080541A

  • Large distributed database clustering systems and methods

    US20130290249A1

Cited By

  • Integrated data migration method and device and storage medium

    CN120910014A