Data processing method and device based on heterogeneous database system
By analyzing the database query log and performing data synchronization and migration tasks, the problem of manual intervention in the existing technology is solved, and the query performance and user experience of heterogeneous database systems are improved.
Patent Information
- Application Number
- CN202010769291.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-03
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-08-03
AI Technical Summary
Existing database queries require manual intervention for optimization, resulting in poor user experience and the inability to automatically handle slow query problems.
By analyzing the data query log of the heterogeneous database system, obtaining pending data tables, determining the target database, and performing data synchronization and migration tasks based on preset read and write scheduling rules, optimizing the data query function.
It realizes automated optimization of data query of heterogeneous database systems, improves user experience, reduces manual intervention, and improves query performance.
Smart Images

Figure CN113760966B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device based on a heterogeneous database system. Background Art
[0002] A database is a warehouse that organizes, stores, and manages data according to data structures. There are many types of databases, from the simplest tables that store various data to large database systems that can store massive amounts of data. Database technology is the core of various information systems such as management information systems, office automation systems, and decision support systems. It is an important technical means for scientific research and decision management. Therefore, how to quickly find the required data from the database is of great significance.
[0003] Existing database queries all query a specified database without considering the query performance and storage cost. Each database query is independent and the query will not change the data storage structure. After a slow query occurs, manual intervention methods are required to optimize the query, resulting in a poor user experience. Summary of the invention
[0004] In view of this, an embodiment of the present invention provides a data processing method and device based on a heterogeneous database system, which can optimize the data query function of the heterogeneous database system, solve the technical problem in the prior art that manual intervention methods are required to optimize queries, and provide a good user experience.
[0005] To achieve the above objective, according to a first aspect of an embodiment of the present invention, a data processing method based on a heterogeneous database system is provided.
[0006] A data processing method based on a heterogeneous database system according to an embodiment of the present invention includes: obtaining a data table to be processed according to a data query log stored in the heterogeneous database system, and determining a target database corresponding to the data table to be processed; generating a data processing task according to the data table to be processed, a storage database corresponding to the data table to be processed, and the target database; and executing the data processing task based on a preset read-write scheduling rule using the data table to be processed, the storage database, and the target database.
[0007] Optionally, obtaining the data table to be processed based on the data query log stored in the heterogeneous database system includes: querying the data query log to obtain query information corresponding to the data table stored in the heterogeneous database system, the query information including: average query response time, query response monitoring indicators, query failure rate, and query frequency; if the query information corresponding to the data table meets the data synchronization condition, determining that the data table is the first data table that needs to be synchronized; if the query information corresponding to the data table meets the data migration condition, determining that the data table is the second data table that needs to be migrated; wherein, the first data table and the second data table are the data tables to be processed.
[0008] Optionally, the query information corresponding to the data table satisfies the data synchronization conditions, including at least one of the following options: the average query response time corresponding to the data table is greater than the preset average query response time, the query response index corresponding to the data table is greater than the preset query response index, the query failure rate corresponding to the data table is greater than the preset query failure rate, and the query frequency corresponding to the data table is greater than the first preset query frequency.
[0009] Optionally, the query information corresponding to the data table satisfies the data migration condition, including: a query frequency corresponding to the data table is less than a second preset query frequency.
[0010] Optionally, determining the target database corresponding to the data table to be processed includes: based on the correspondence between query information and database type, determining the first target database that needs to synchronize data with the first data table according to the query information corresponding to the first data table; based on the correspondence between database type and storage cost, determining the second target database that needs to migrate data with the second data table and the migration data corresponding to the second data table according to the storage information corresponding to the second data table, the storage information including: storage database, storage time range, and storage cost.
[0011] Optionally, the data processing task is performed using the data table to be processed, the storage database and the target database, including: synchronizing the first data table from a first storage database corresponding to the first data table to the first target database according to the table structure corresponding to the first target database; migrating the migration data corresponding to the second data table from a second storage database corresponding to the second data table to the second target database according to the table structure corresponding to the second target database.
[0012] Optionally, after executing the data processing task, the method further includes: performing data verification on the data table to be processed and the target data table corresponding to the data processing task; if the verification passes, updating the storage information in the data dictionary according to the data table to be processed and the target data table, and updating the query information in the data dictionary using the data query log; if the verification fails, re-executing the data processing task.
[0013] Optionally, the method also includes: receiving a data query request, obtaining input parameters corresponding to the data query request, the input parameters including: a data table to be queried, a query dimension, and a query condition; using the storage information and query information in the data dictionary to query the database to be queried corresponding to the data table to be queried; generating a data query task corresponding to the database to be queried according to the data table to be queried, the query dimension, and the query condition; and executing the data query task based on preset read and write scheduling rules.
[0014] Optionally, the preset read-write scheduling rules include at least one of the following options: when there is a data query task, execute the data query task first and then execute the data processing task; when there is no data query task, execute the data processing task directly; when the data table to be processed corresponding to the data processing task and the data table to be queried corresponding to the data query task are the same, execute the data processing task first and then execute the data query task.
[0015] Optionally, the method further includes: generating a data query log corresponding to the data query request; and updating the data dictionary using the generated data query log.
[0016] To achieve the above objective, according to a second aspect of an embodiment of the present invention, a data processing device based on a heterogeneous database system is provided.
[0017] A data processing device based on a heterogeneous database system according to an embodiment of the present invention includes: a determination module, which is used to obtain a data table to be processed based on a data query log stored in the heterogeneous database system, and determine a target database corresponding to the data table to be processed; a generation module, which is used to generate a data processing task based on the data table to be processed, a storage database corresponding to the data table to be processed, and the target database; and an execution module, which is used to execute the data processing task based on a preset read-write scheduling rule using the data table to be processed, the storage database, and the target database.
[0018] Optionally, the determination module is also used to: query the data query log to obtain query information corresponding to the data table stored in the heterogeneous database system, the query information including: average query response time, query response monitoring indicators, query failure rate, and query frequency; if the query information corresponding to the data table meets the data synchronization conditions, determine that the data table is the first data table that needs to be synchronized; if the query information corresponding to the data table meets the data migration conditions, determine that the data table is the second data table that needs to be migrated; wherein, the first data table and the second data table are the data tables to be processed.
[0019] Optionally, the query information corresponding to the data table satisfies the data synchronization conditions, including at least one of the following options: the average query response time corresponding to the data table is greater than the preset average query response time, the query response index corresponding to the data table is greater than the preset query response index, the query failure rate corresponding to the data table is greater than the preset query failure rate, and the query frequency corresponding to the data table is greater than the first preset query frequency.
[0020] Optionally, the query information corresponding to the data table satisfies the data migration condition, including at least one of the following options: a query failure rate corresponding to the data table is greater than a second preset query failure rate, and a query frequency corresponding to the data table is less than a second preset query frequency.
[0021] Optionally, the determination module is also used to: based on the correspondence between query information and database type, determine the first target database that needs to synchronize data with the first data table according to the query information corresponding to the first data table; based on the correspondence between database type and storage cost, determine the second target database that needs to migrate data with the second data table and the migration data corresponding to the second data table according to the storage information corresponding to the second data table, the storage information including: storage database, storage time range, and storage cost.
[0022] Optionally, the execution module is also used to: synchronize the first data table from the first storage database corresponding to the first data table to the first target database according to the table structure corresponding to the first target database; and migrate the migration data corresponding to the second data table from the second storage database corresponding to the second data table to the second target database according to the table structure corresponding to the second target database.
[0023] Optionally, the execution module is also used to: perform data verification on the data table to be processed and the target data table corresponding to the data processing task; if the verification passes, update the storage information in the data dictionary according to the data table to be processed and the target data table, and update the query information in the data dictionary using the data query log; if the verification fails, re-execute the data processing task.
[0024] Optionally, the device also includes a query module, which is used to: receive a data query request, obtain input parameters corresponding to the data query request, and the input parameters include: a data table to be queried, a query dimension, and a query condition; use the storage information and query information in the data dictionary to query the database to be queried corresponding to the data table to be queried; generate a data query task corresponding to the database to be queried according to the data table to be queried, the query dimension, and the query condition; execute the data query task based on preset read and write scheduling rules.
[0025] Optionally, the preset read-write scheduling rules include at least one of the following options: when there is a data query task, execute the data query task first and then execute the data processing task; when there is no data query task, execute the data processing task directly; when the data table to be processed corresponding to the data processing task and the data table to be queried corresponding to the data query task are the same, execute the data processing task first and then execute the data query task.
[0026] Optionally, the query module is further used to: generate a data query log corresponding to the data query request; and update the data dictionary using the generated data query log.
[0027] To achieve the above objective, according to a third aspect of an embodiment of the present invention, an electronic device is provided.
[0028] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method based on the heterogeneous database system according to an embodiment of the present invention.
[0029] To achieve the above objective, according to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided.
[0030] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a data processing method based on a heterogeneous database system according to an embodiment of the present invention is implemented.
[0031] An embodiment of the above invention has the following advantages or beneficial effects: it is able to obtain the data table to be processed by querying the stored data query log, and then generate a data processing task in combination with the storage database and the target database corresponding to the data table to be processed. Finally, it is able to execute the data processing task based on the preset read and write scheduling rules. It is able to automatically perform data processing on the data table to be processed through the data query log, thereby optimizing the data query function of the heterogeneous database system, solving the technical problem in the prior art that manual intervention methods are required to optimize queries, and providing a good user experience.
[0032] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0034] Figure 1 is a schematic diagram of main steps of a data processing method based on a heterogeneous database system according to an embodiment of the present invention;
[0035] Figure 2 is a schematic diagram of the main process of a data synchronization method based on a heterogeneous database system according to an embodiment of the present invention;
[0036] Figure 3 is a schematic diagram of the main process of a data migration method based on a heterogeneous database system according to an embodiment of the present invention;
[0037] Figure 4 is a schematic diagram of the main process of a data query method based on a heterogeneous database system according to an embodiment of the present invention;
[0038] Figure 5 is a schematic diagram of the structure of a heterogeneous database system according to an embodiment of the present invention;
[0039] Figure 6 is a schematic diagram of main modules of a data processing device based on a heterogeneous database system according to an embodiment of the present invention;
[0040] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0041] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0043] Figure 1 FIG. 1 is a schematic diagram of the main steps of a data processing method based on a heterogeneous database system according to an embodiment of the present invention. Figure 1 As shown, the main steps of the data processing method based on the heterogeneous database system may include steps S101 to S103.
[0044] Step S101: obtaining a data table to be processed according to a data query log stored in a heterogeneous database system, and determining a target database corresponding to the data table to be processed;
[0045] Among them, the heterogeneous database system is a collection of multiple related database systems, which can realize data sharing and transparent access. Each database system already exists before joining the heterogeneous database system and has its own management software. The heterogeneous database system of the embodiment of the present invention can include different types of databases, such as MySql (i.e., relational database management system), MongoDB (i.e., distributed document storage database), HDFS (i.e., distributed file system, suitable for large-scale data sets), HBase (i.e., distributed, column-oriented open source database), Redis (i.e., key-value storage system), etc. MySql belongs to relational database, MongoDB and HDFS belong to document database, HBase belongs to column storage database, and Redis belongs to cache.
[0046] The heterogeneous database system in the embodiment of the present invention may include multiple types of databases, and each database stores a data table. A user may access the heterogeneous database system, query the data tables stored in the heterogeneous database system, obtain the required data information, and generate a query record of the data table, i.e., a data query log. In the embodiment of the present invention, the query record of each data table in the heterogeneous database system may be analyzed to determine whether each data table needs data synchronization or data migration. Specifically, if the query performance of a data table is poor, such as the query response time of data table A1 is too long, it means that data table A1 needs data synchronization, so that the query response time of data table A1 can be reduced and the performance of the heterogeneous database system can be improved. If the storage cost of a data table is high, such as the storage cost of data table A2 is high, data migration can be performed on data table A2, and a database with low storage cost can be used for storage. In the embodiment of the present invention, the data table to be processed may include a data table that needs data synchronization and a data table that needs data migration, and the data table to be processed can be obtained through the data query log.
[0047] Step S102: Generate a data processing task according to the to-be-processed data table, the storage database corresponding to the to-be-processed data table, and the target database.
[0048] After obtaining the data table to be processed, it is also necessary to determine the target database corresponding to the data table to be processed. The target database refers to the database that needs to be processed. For example, if data table A1 needs to be synchronized to database M10, then database M10 is the target database corresponding to data table A1, or if data table A2 needs to be migrated to database M8, then database M8 is the target database corresponding to data table A2.
[0049] After obtaining the data table to be processed and the target database corresponding to the data table to be processed, a data processing task can be generated in combination with the storage database corresponding to the data table to be processed. Among them, the data processing task may include a data synchronization task and a data migration task. If the data table to be processed is a data table that needs data synchronization, the generated data processing task is a data synchronization task. If the data table to be processed is a data table that needs data migration, the generated data processing task is a data migration task. In addition, the storage database corresponding to the data table to be processed refers to the database that currently stores the data table to be processed. For example, if the storage database corresponding to the data table A1 is M1, it means that the data table A1 is currently stored in the storage database M1.
[0050] Step S103: Based on the preset read-write scheduling rules, the data table to be processed, the storage database and the target database are used to execute the data processing task.
[0051] After generating the data processing task, the data processing task can be executed based on the preset read-write scheduling rules, using the data table to be processed, the storage database and the target database. For example, a data synchronization task is generated based on the data table A1 that needs to be synchronized, the corresponding storage database M1, and the corresponding target database M10, and the data table A1 can be synchronized from the storage database M1 to the target database M10; a data migration task is generated based on the data table A2 that needs to be migrated, the corresponding storage database M2, and the corresponding target database M8, and the data table A2 can be migrated from the storage database M2 to the target database M8.
[0052] For heterogeneous database systems, data synchronization is to synchronize data tables from one database to another, and data migration is to migrate data tables from one database to another. Therefore, data synchronization and data migration are both write operations. Considering that heterogeneous database systems can also provide data query functions, and data query is to read data tables from the database, which is a read operation. Pre-setting read-write scheduling rules, that is, pre-setting the execution order of data synchronization, data migration, and data query, can avoid read-write conflicts and ensure that the heterogeneous database system can provide good services.
[0053] In an embodiment of the present invention, the preset read-write scheduling rules may include at least one of the following options: when there is a data query task, execute the data query task first and then execute the data processing task; when there is no data query task, execute the data processing task directly; when the data table to be processed corresponding to the data processing task and the data table to be queried corresponding to the data query task are the same, execute the data processing task first and then execute the data query task.
[0054] Specifically, when there is a read operation, the write operation is not performed. That is, when there is a data query task, the data query task needs to be performed first. After the data query task is completed, the data processing task is performed. For example, if there is a data query task D1, a data synchronization task D2, and a data migration task D3, the data query task D1 needs to be performed first, and then the data synchronization task D2 and the data migration task D3. In addition, D2 and D3 can be performed at the same time, or D2 can be performed first, and then D3. There is no limitation for comparison.
[0055] When executing a write operation, if there is a read operation at this time, if the write operation and the read operation correspond to the same data table, the read operation will not be executed first. If the write operation and the read operation do not correspond to the same data table, the read operation can be executed, that is, when executing a data processing task, if there is a data query task at this time, if the data table corresponding to the data processing task and the data query task is the same, the data query task will not be executed first. If the data table corresponding to the data processing task and the data query task is different, the data query task can be executed at this time. For example, when executing the data synchronization task D2, there is a data query task D1. At this time, it is determined whether D2 and D1 correspond to the same data table. If so, D1 can be executed only after D2 is executed. If not, D1 can be executed. In addition, considering that when there is a read operation, the write operation is not executed, then the execution of D2 can be suspended, and then the execution of D2 can be continued after the execution of D1 is completed.
[0056] The data processing technology solution based on the heterogeneous database system in the embodiment of the present invention obtains the data table to be processed by querying the stored data query log, and then generates a data processing task in combination with the storage database and the target database corresponding to the data table to be processed. Finally, the data processing task can be executed based on the preset read and write scheduling rules. The data query log can be used to automatically perform data processing on the data table to be processed, thereby optimizing the data query function of the heterogeneous database system, solving the technical problem in the prior art that manual intervention methods are required to optimize queries, and providing a good user experience.
[0057] The data processing method based on the heterogeneous database system can automatically perform data processing on the data table to be processed, so the acquisition of the data table to be processed is an important component. In the reference embodiment of the present invention, according to the data query log stored in the heterogeneous database system, the data table to be processed is obtained, which may include:
[0058] Step S1011, querying the data query log to obtain query information corresponding to the data table stored in the heterogeneous database system;
[0059] Step S1012: if the query information corresponding to the data table meets the data synchronization condition, determine that the data table is the first data table that needs to be synchronized;
[0060] Step S1013: if the query information corresponding to the data table meets the data migration condition, determine that the data table is a second data table that needs to be migrated.
[0061] In step S1011, the data query log is queried to obtain query information corresponding to each data table in the heterogeneous database system, wherein the query information may include: average query response time, query response monitoring index, query failure rate, and query frequency. The average query response time refers to the average response time for querying a data table, for example, the average response time for querying data table A1 within one day. The query response monitoring indicators can be TP99, TP90, etc. TP is Top Percentile, which is the same as the average and median. It is a term in statistics. TP99 refers to the minimum time required to ensure that 99% of data query requests can be responded to, and TP90 refers to the minimum time required to ensure that 90% of data query requests can be responded to. For example, if the query request response times of 100 data table A1 are sorted from small to large, the response time ranked 99th is TP99, and the response time ranked 90th is TP90. The query failure rate refers to the failure rate of data query requests. For example, there are 100 query requests for data table A1, of which 85 requests can query the required data, and the query failure rate is 15%. The query frequency refers to the number of times a data table is queried per unit time, for example, the number of times data table A1 is queried within one day.
[0062] After obtaining the query information of each data table, each data table can be analyzed to determine whether the data table is a data table to be processed. In an embodiment of the present invention, the data table to be processed may include a first data table that needs to be synchronized and a second data table that needs to be migrated. The following is a specific method for determining the data table to be processed.
[0063] (1) If the query information corresponding to the data table meets the data synchronization condition, the data table is determined to be the first data table that needs to be synchronized.
[0064] In an embodiment of the present invention, the query information corresponding to the data table satisfies the data synchronization condition and may include at least one of the following options: (1) the average query response time corresponding to the data table is greater than the preset average query response time; (2) the query response index corresponding to the data table is greater than the preset query response index; (3) the query failure rate corresponding to the data table is greater than the preset query failure rate; (4) the query frequency corresponding to the data table is greater than the first preset query frequency. Each data table in the heterogeneous database system is analyzed. If a data table satisfies at least one of the above conditions (1) to (4), it means that the query performance corresponding to the data table is not good, so it can be determined that the data table is the first data table that needs to be synchronized.
[0065] Among them, the preset query average response time can be set according to time or number of times. For example, if the preset query average response time is set according to time, a certain time can be directly set as the preset query average response time based on historical experience or specific business; if the preset query average response time is set according to the number of times, the query average response time of the data table can be sorted from large to small, and the top N data tables are selected as the data tables that need to be synchronized. Correspondingly, the preset query response index, the preset query failure rate, and the first preset query frequency can also be set according to time or number of times, which is not described in detail in the embodiment of the present invention.
[0066] (ii) If the query information corresponding to the data table meets the data migration condition, the data table is determined to be the second data table that needs to be migrated. In an embodiment of the present invention, the query information corresponding to the data table meets the data migration condition and may include: the query frequency corresponding to the data table is less than the second preset query frequency. Each data table in the heterogeneous database system is analyzed. If the query frequency corresponding to a data table is less than the second preset query frequency, it means that the query frequency corresponding to the data table is low. For example, if the data recorded in the data table is data from 1 year ago, the query frequency is low. Then the data table can be migrated to a database with low storage cost. Therefore, it can be determined that the data table is the second data table that needs to be migrated.
[0067] In the data processing method based on a heterogeneous database system of an embodiment of the present invention, it is possible to directly analyze the query information of the data table to obtain the data table that needs data synchronization and data migration, so that data synchronization tasks and data migration tasks can be executed, thereby achieving the effect of automatically optimizing the data table storage of the heterogeneous database system, thereby improving the query performance of the heterogeneous database system.
[0068] After obtaining the data tables that need to be synchronized and the data tables that need to be migrated, the corresponding target database can be determined. The target database has been explained in step S102 above and will not be repeated here. In a reference embodiment of the present invention, determining the target database corresponding to the data table to be processed may include: determining the first target database corresponding to the first data table using step S1021, and determining the second target database corresponding to the second data table using step S1022.
[0069] Step S1021: Based on the correspondence between the query information and the database type, and according to the query information corresponding to the first data table, determine a first target database for which data synchronization of the first data table is required.
[0070] The database type refers to the specific type of the database. For example, MySql belongs to a relational database, MongoDB and HDFS belong to a document database, HBase belongs to a column storage database, and Redis belongs to a cache. For a data table, a certain type of database can be determined as the optimal storage database based on the query information of the data table.
[0071] In the embodiment of the present invention, the correspondence between query information and database type defines the optimal database type corresponding to different query information, which may be: (1) for a data table with a high query frequency and a small number of table rows, the corresponding optimal database is the relational database MySql; (2) for a data table with a low query frequency and a large number of table rows, the optimal storage method may be to store it in the document-type database HDFS in the Orc format (i.e., a file storage format); (3) for a data table with a large number of query conditions, the optimal storage method may be to store it in the document-type database HDFS in the Orc format; (4) for a data table with a high query frequency, an aggregation dimension and generalization to a result table, the corresponding optimal database is the relational database MySql; (5) for a data table with a long average query response time and generalization to a result table, the optimal storage method may be to store it in the document-type database HDFS in the Orc format; (6) for a data table with a small generalized result table and a high query frequency, the corresponding optimal database is the cache Redis. Since the data table involves tens of thousands of records, the number of data queries can be greatly reduced by using aggregation techniques such as summarization, average, and extreme values. Aggregation dimensions refer to the dimensions involved in aggregation techniques; query conditions refer to the specific conditions set when querying data in a data table. In addition, both aggregation dimensions and query conditions can be regarded as query information of a data table, and a data table can be queried according to specific aggregation dimensions and query conditions. Data generalization is an analysis process from relatively low-level concepts to higher-level concepts and an abstract overview of a large amount of task-related data in the database. Generalization to a result table refers to the existence of a result table in the data table that generalizes some data.
[0072] After determining the query information corresponding to the first data table, the optimal database storing the first data table can be obtained based on the correspondence between the query information and the database type, so the optimal database can be directly determined as the first target database. In addition, it should be noted that if, based on the correspondence between the query information and the database type, it is known that the database currently storing the first data table is already the optimal database, in this case, the data synchronization task may not be performed on the first data table.
[0073] Step S1022: Based on the correspondence between database type and storage cost, and according to the storage information corresponding to the second data table, determine the second target database for data migration of the second data table and the migration data corresponding to the second data table. The storage information may include: storage database, storage time range, and storage cost.
[0074] The corresponding relationship between database type and storage cost sets the storage cost corresponding to different database types. The storage cost is equivalent to the amount of memory occupied. For example, the storage cost required for storing data table A1 in relational database MySql is C1, the storage cost required for storing data table A1 in document database MongoDB is C2, the storage cost required for storing data table A1 in column storage database HBase is C3, and the storage cost required for storing data table A1 in cache Redis is C4. Since the query frequency corresponding to the second data table is low, the second data table can be migrated to a database with lower storage cost.
[0075] In the technical solution of the embodiment of the present invention, the first target database and the second target database can be determined based on the correspondence between query information and database type and the correspondence between database type and storage cost, and the data table storage of the heterogeneous database system can be automatically optimized, thereby solving the technical problem that the prior art requires manual data optimization and further improving the query performance of the heterogeneous database system.
[0076] In the reference embodiment of the present invention, using the data table to be processed, the storage database and the target database to perform the data processing task may include: step S1031, according to the table structure corresponding to the first target database, synchronizing the first data table from the first storage database corresponding to the first data table to the first target database; step S1032, according to the table structure corresponding to the second target database, migrating the migration data corresponding to the second data table from the second storage database corresponding to the second data table to the second target database. Wherein, the migration data refers to the data that needs to be migrated from the second data table to the second target database.
[0077] Different databases use different table structures to store data tables. Therefore, if you need to use the target database for data synchronization or data migration, you need to generate the table structure of the target database first, and then perform data synchronization or data migration tasks according to the generated table structure. The table structure can be a schema, and you can use the open source tool Sqoop to perform data synchronization. Sqoop is mainly used to transfer data between Hadoop (i.e., the basic framework of distributed systems, HDFS is one of its cores) and traditional databases. It can import data from a relational database into HDFS, and it can also import data from HDFS into a relational database.
[0078] For data synchronization tasks, the table structure corresponding to the first target database is generated, and then the first data table is synchronized to the first target data table of the first target database according to the table structure corresponding to the first target database; for data migration tasks, the table structure corresponding to the second target database is generated, and then the migration data corresponding to the second data table is migrated to the second target data table of the second target database according to the table structure corresponding to the second target database. In addition, for data migration tasks, the migration data corresponding to the second data table can be first synchronized to the second target data table, and then the migration data can be deleted from the second data table.
[0079] In addition, after executing the data processing task, it is necessary to verify the data processing task and update the data table information stored in the heterogeneous database system. Therefore, in the reference embodiment of the present invention, after executing the data processing task, the data processing method based on the heterogeneous database system may also include: performing data verification on the data table to be processed and the target data table corresponding to the data processing task; if the verification passes, updating the storage information in the data dictionary according to the data table to be processed and the target data table, and updating the query information in the data dictionary using the data query log; if the verification fails, re-executing the data processing task. The target data table can be a data table corresponding to a data synchronization task or a data migration task.
[0080] Assume that data table A1 needs to be synchronized to database M10. Then, a new data table A10 needs to be added to M10, the data in A1 needs to be synchronized to A10, and A10 needs to be defined as the target data table. After executing the data synchronization task, the data in A1 and A10 are verified for consistency to verify whether the data in A1 and A10 are consistent. If the verification passes, the data synchronization task can be considered successful, and the information of A10 in the data dictionary can be updated. In addition, the query records of A1 can be extracted from the data query log, and A10 can be queried to obtain the query test results of A10, and the query information of A10 in the data dictionary can be updated using the query test results. If the verification fails, the data synchronization task needs to be re-executed.
[0081] For another example, if you need to migrate the data in the time range t1 to t2 in data table A2 to database M8, you need to add a new data table A8 in M8, migrate the data in the time range t1 to t2 in A2 to A8, and define A8 as the target data table. After executing the data migration task, verify whether the data in the time range t1 to t2 is cleared from A2, and verify whether A8 contains the data in the time range t1 to t2 in A2. If the verification passes, the data synchronization task can be considered successful, and the information of A8 in the data dictionary can be updated. In addition, the data query records in the time range t1 to t2 in A2 can be extracted from the data query log, and A8 can be queried to obtain the query test results of A8, and the query information of A8 in the data dictionary can be updated using the query test results. If the verification fails, the data migration task needs to be re-executed.
[0082] Figure 2 Schematic diagram of the main process of the data synchronization method based on a heterogeneous database system according to an embodiment of the present invention. Figure 2 As shown, the main process of the data synchronization method based on the heterogeneous database system may include:
[0083] Step S201, querying the data query log to obtain query information corresponding to the data table stored in the heterogeneous database system;
[0084] Step S202: if the query information corresponding to the data table meets the data synchronization condition, determine that the data table is the first data table that needs to be synchronized;
[0085] Step S203, based on the correspondence between the query information and the database type, and according to the query information corresponding to the first data table, determining a first target database for which data synchronization of the first data table is required;
[0086] Step S204, generating a data synchronization task according to the first data table, the storage database corresponding to the first data table, and the first target database;
[0087] Step S205, based on a preset read-write scheduling rule and in accordance with a table structure corresponding to the first target database, synchronizing the first data table from the first storage database corresponding to the first data table to the first target data table of the first target database;
[0088] Step S206, performing data consistency verification on the first data table and the first target data table;
[0089] Step S207, if the verification is successful, then updating the storage information in the data dictionary according to the first data table and the first target data table, and updating the query information in the data dictionary using the data query log;
[0090] Step S208: If the verification fails, the data synchronization task is re-executed.
[0091] Among them, the query information corresponding to the data table satisfies the data synchronization conditions and may include at least one of the following options: the average query response time corresponding to the data table is greater than the preset average query response time, the query response index corresponding to the data table is greater than the preset query response index, the query failure rate corresponding to the data table is greater than the preset query failure rate, and the query frequency corresponding to the data table is greater than the first preset query frequency.
[0092] Figure 3 FIG. 1 is a schematic diagram of the main process of a data migration method based on a heterogeneous database system according to an embodiment of the present invention. Figure 3 As shown in the figure, the main process of the data migration method based on the heterogeneous database system may include:
[0093] Step S301, querying the data query log to obtain query information corresponding to the data table stored in the heterogeneous database system;
[0094] Step S302: if the query information corresponding to the data table meets the data migration condition, determine that the data table is a second data table that needs to be migrated;
[0095] Step S303, based on the correspondence between the database type and the storage cost and according to the storage information corresponding to the second data table, determining the second target database for which data migration of the second data table is required and the migration data corresponding to the second data table;
[0096] Step S304, generating a data migration task according to the second data table, the storage database corresponding to the second data table, and the second target database;
[0097] Step S305, based on the preset read-write scheduling rule and in accordance with the table structure corresponding to the second target database, migrate the migration data from the second storage database corresponding to the second data table to the second target data table of the second target database;
[0098] Step S306, verifying the second data table and the second target data table;
[0099] Step S307, if the verification is successful, then updating the storage information in the data dictionary according to the second data table and the second target data table, and updating the query information in the data dictionary using the data query log;
[0100] Step S308: If the verification fails, the data migration task is re-executed.
[0101] The query information corresponding to the data table satisfies the data migration condition, which may include: the query frequency corresponding to the data table is less than the second preset query frequency.
[0102] A heterogeneous database system can provide a data query function. Therefore, in a reference embodiment of the present invention, a data processing method based on a heterogeneous database system may further include: receiving a data query request and obtaining input parameters corresponding to the data query request; using the storage information and query information in the data dictionary to query the database to be queried corresponding to the data table to be queried; generating a data query task corresponding to the database to be queried according to the data table to be queried, the query dimension, and the query condition; and executing the data query task based on preset read and write scheduling rules.
[0103] The data table to be queried, the query dimension, and the query condition are the input parameters corresponding to the data query request. Specifically, you can enter the unique identifier of the data table to be queried, such as the table name table1; the query dimension can be the city dimension, season dimension, etc. of the data; the query condition refers to the specific query condition, such as querying the data on January 1, 2020.
[0104] The data dictionary records the query information and storage information of each data table. For example, for data table A1, the data dictionary records the database storing A1, the query conditions corresponding to A1, the aggregation dimensions corresponding to A1, the average response time for querying A1, the response monitoring indicators for querying A1, the failure rate of querying A1, the frequency of querying A1, the number of table rows of A1, and the update time of A1. The data dictionary also records the storage date range and storage cost of A1.
[0105] Therefore, after determining the data table to be queried, the database storing the data table to be queried can be determined based on the information stored in the data dictionary, and then the database to be queried can be determined by the query performance of the determined database, wherein the query performance can be determined by setting the weights of the average query response time, query response monitoring indicators, query failure rate and query frequency, and then calculating the query performance of each determined database, and selecting the database with the best query performance as the database to be queried. Then, using the data table to be queried, the query dimension, and the query condition, the data query task corresponding to the database to be queried is generated, and finally the generated data query task can be executed based on the preset read and write scheduling rules. Among them, the preset scheduling read and write rules have been explained in the above step S101, and will not be repeated here.
[0106] The data query task may be a database query statement. The format of the database query statement is different for different types of databases. Therefore, after determining the database to be queried, the query statement corresponding to the database to be queried is generated using the data table to be queried, the query dimension, and the query condition.
[0107] In addition, in the reference embodiment of the present invention, the data processing method based on the heterogeneous database system may further include: generating a data query log corresponding to the data query request; and updating the data dictionary using the generated data query log. The embodiment of the present invention can determine the data table to be processed by querying the data query log. Therefore, after receiving the data query request, it is necessary to generate a data query log corresponding to the data query request, and use the data query log to update the query information and storage information stored in the data dictionary.
[0108] Figure 4 FIG. 1 is a schematic diagram of the main process of a data query method based on a heterogeneous database system according to an embodiment of the present invention. Figure 4 As shown, the main process of the data query method based on the heterogeneous database system may include:
[0109] Step S401, receiving a data query request, and obtaining input parameters corresponding to the data query request, where the input parameters may include: a data table to be queried, a query dimension, and a query condition;
[0110] Step S402, using the storage information and query information in the data dictionary, querying the database to be queried corresponding to the data table to be queried;
[0111] Step S403, generating a data query task corresponding to the database to be queried according to the data table to be queried, the query dimension, and the query condition;
[0112] Step S404, executing the data query task based on the preset read-write scheduling rules;
[0113] Step S405: Generate a data query log corresponding to the data query request, and use the generated data query log to update the data dictionary.
[0114] In the embodiment of the present invention, a database with good query performance can be selected for data query according to the storage information and query information of the data dictionary, thereby improving data query efficiency and providing a good user experience.
[0115] The following is a detailed description of the structure of a heterogeneous database system. Figure 5 Schematic diagram of the structure of a heterogeneous database system according to an embodiment of the present invention. Figure 5 As shown, the heterogeneous database system 500 may include: a data query interface 501, a data dictionary 502, a data query log component 503, a data processing task generation component 504, a data query statement component 505, a read-write scheduling control component 506, and different types of databases 507.
[0116] The data query interface 501 is used to receive a data processing request and determine the input parameters according to the data processing request, wherein the input parameters may include: the data table to be queried, the query dimension, and the query condition; and is used to configure database connections with different types of databases. The data dictionary 502 is used to store the query information and storage information of the data table; and is also used to determine the database to be queried corresponding to the data table to be queried according to the query information and storage information.
[0117] The data query log component 503 is used to collect query records of data tables regularly or in real time, and use the query records to update the query information and storage information of the data dictionary; it is also used to obtain data tables that need data synchronization and data tables that need data migration based on the data query log regularly or in real time.
[0118] The data processing task generation component 504 is used to determine the target database corresponding to the data table that needs to be synchronized based on the corresponding relationship between the query information and the database type, and to add a target data table according to the determined table structure of the target database; it is also used to determine the target database corresponding to the data table that needs to be migrated based on the corresponding relationship between the database type and the storage cost, and to add a target data table according to the determined table structure of the target database. The data processing task generation component 504 is also used to verify the data processing task and update the data table information stored in the heterogeneous database system.
[0119] The data query statement component 505 is used to generate a query statement corresponding to the database to be queried using the data table to be queried, the query dimension, and the query condition. The read-write scheduling control component 506 is used to execute data synchronization tasks, data migration tasks, and data query tasks based on preset read-write scheduling rules. Different types of databases 507 may include MySql, MongoDB, HDFS, HBase, Redis and other databases.
[0120] The heterogeneous database system of the embodiment of the present invention can obtain the data table to be processed by querying the stored data query log, and then generate a data processing task in combination with the storage database and the target database corresponding to the data table to be processed. Finally, the data processing task can be executed based on the preset read and write scheduling rules. The data query log can be used to automatically perform data processing on the data table to be processed, thereby optimizing the data query function of the heterogeneous database system, solving the technical problem in the prior art that manual intervention methods are required to optimize queries, and providing a good user experience.
[0121] Figure 6 Schematic diagram of main modules of a data processing device based on a heterogeneous database system according to an embodiment of the present invention. Figure 6As shown, the main modules of the data processing device 600 based on the heterogeneous database system may include: a determination module 601 , a generation module 602 and an execution module 603 .
[0122] Among them, the determination module 601 can be used to obtain the data table to be processed and determine the target database corresponding to the data table to be processed based on the data query log stored in the heterogeneous database system; the generation module 602 can be used to generate data processing tasks based on the data table to be processed, the storage database corresponding to the data table to be processed, and the target database; the execution module 603 can be used to execute data processing tasks based on the preset read and write scheduling rules, using the data table to be processed, the storage database and the target database.
[0123] In an embodiment of the present invention, the determination module 601 can also be used to: query the data query log to obtain the query information corresponding to the data table stored in the heterogeneous database system; if the query information corresponding to the data table meets the data synchronization condition, determine that the data table is the first data table that needs to be synchronized; if the query information corresponding to the data table meets the data migration condition, determine that the data table is the second data table that needs to be migrated. The query information may include: average query response time, query response monitoring indicators, query failure rate, and query frequency; the first data table and the second data table are data tables to be processed.
[0124] In an embodiment of the present invention, the query information corresponding to the data table satisfies the data synchronization conditions and may include at least one of the following options: the average query response time corresponding to the data table is greater than the preset average query response time, the query response index corresponding to the data table is greater than the preset query response index, the query failure rate corresponding to the data table is greater than the preset query failure rate, and the query frequency corresponding to the data table is greater than the first preset query frequency.
[0125] In an embodiment of the present invention, the query information corresponding to the data table satisfies the data migration condition and may include at least one of the following options: the query failure rate corresponding to the data table is greater than the second preset query failure rate, and the query frequency corresponding to the data table is less than the second preset query frequency.
[0126] In the embodiment of the present invention, the determination module 601 may also be used to: determine the first target database that needs to perform data synchronization on the first data table based on the correspondence between the query information and the database type, according to the query information corresponding to the first data table; determine the second target database that needs to perform data migration on the second data table and the migration data corresponding to the second data table based on the correspondence between the database type and the storage cost, according to the storage information corresponding to the second data table. The storage information may include: storage database, storage time range, and storage cost.
[0127] In an embodiment of the present invention, the execution module 603 can also be used to: synchronize the first data table from the first storage database corresponding to the first data table to the first target database according to the table structure corresponding to the first target database; and migrate the migration data corresponding to the second data table from the second storage database corresponding to the second data table to the second target database according to the table structure corresponding to the second target database.
[0128] In an embodiment of the present invention, the execution module 603 can also be used to: perform data verification on the data table to be processed and the target data table corresponding to the data processing task; if the verification passes, update the storage information in the data dictionary according to the data table to be processed and the target data table, and update the query information in the data dictionary using the data query log; if the verification fails, re-execute the data processing task.
[0129] In an embodiment of the present invention, a data processing device based on a heterogeneous database system may further include: a query module (not shown in the figure). The query module may be used to: receive a data query request, obtain an input parameter corresponding to the data query request; use the storage information and query information in the data dictionary to query the database to be queried corresponding to the data table to be queried; generate a data query task corresponding to the database to be queried according to the data table to be queried, the query dimension, and the query condition; execute the data query task based on the preset read and write scheduling rules. The input parameter may include: the data table to be queried, the query dimension, and the query condition.
[0130] In an embodiment of the present invention, the preset read-write scheduling rules may include at least one of the following options: when there is a data query task, execute the data query task first and then execute the data processing task; when there is no data query task, execute the data processing task directly; when the data table to be processed corresponding to the data processing task and the data table to be queried corresponding to the data query task are the same, execute the data processing task first and then execute the data query task.
[0131] In the embodiment of the present invention, the query module may also be used to: generate a data query log corresponding to the data query request; and update the data dictionary using the generated data query log.
[0132] From the above description, it can be seen that the data processing device based on the heterogeneous database system of the embodiment of the present invention can obtain the data table to be processed by querying the stored data query log, and then generate a data processing task in combination with the storage database and the target database corresponding to the data table to be processed. Finally, the data processing task can be executed based on the preset read and write scheduling rules. It can automatically perform data processing on the data table to be processed through the data query log, thereby optimizing the data query function of the heterogeneous database system, solving the technical problem in the prior art that manual intervention methods are required to optimize queries, and providing a good user experience.
[0133] Figure 7 An exemplary system architecture 700 is shown to which a data processing method based on a heterogeneous database system or a data processing device based on a heterogeneous database system according to an embodiment of the present invention can be applied.
[0134] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0135] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).
[0136] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0137] The server 705 may be a server that provides various services, such as a backend management server (only for example) that supports shopping websites browsed by users using the terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only for example) to the terminal device.
[0138] It should be noted that the data processing method based on the heterogeneous database system provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data processing device based on the heterogeneous database system is generally arranged in the server 705 .
[0139] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0140] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0141] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0142] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.
[0143] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.
[0144] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0146] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, they may be described as: a processor includes a determination module, a generation module, and an execution module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the determination module may also be described as "a module for obtaining a data table to be processed and determining a target database corresponding to the data table to be processed based on a data query log stored in a heterogeneous database system."
[0147] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a data table to be processed according to a data query log stored in a heterogeneous database system, and determining a target database corresponding to the data table to be processed; generating a data processing task according to the data table to be processed, the storage database corresponding to the data table to be processed, and the target database; and executing the data processing task based on the preset read-write scheduling rules, using the data table to be processed, the storage database, and the target database.
[0148] According to the technical solution of the embodiment of the present invention, by querying the stored data query log, the data table to be processed is obtained, and then the data processing task is generated in combination with the storage database and the target database corresponding to the data table to be processed. Finally, the data processing task can be executed based on the preset read and write scheduling rules. Through the data query log, data processing can be automatically performed on the data table to be processed, thereby optimizing the data query function of the heterogeneous database system, solving the technical problem in the prior art that manual intervention methods are required to optimize queries, and providing a good user experience.
[0149] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A data processing method based on a heterogeneous database system, characterized in that: include: Obtaining a data table to be processed according to a data query log stored in a heterogeneous database system, and determining a target database corresponding to the data table to be processed; Generate a data processing task according to the to-be-processed data table, the storage database corresponding to the to-be-processed data table and the target database; Based on the preset read-write scheduling rules, the data table to be processed, the storage database and the target database are used to execute the data processing task; wherein, after executing the data processing task, data verification is performed on the data table to be processed and the target data table corresponding to the data processing task; if the verification is passed, the storage information in the data dictionary is updated according to the data table to be processed and the target data table, and the query information in the data dictionary is updated using the data query log; Receive a data query request, and obtain input parameters corresponding to the data query request, wherein the input parameters include: a data table to be queried, a query dimension, and a query condition; Using the storage information and query information in the data dictionary, query the database to be queried corresponding to the data table to be queried; wherein, the database storing the data table to be queried is determined according to the data dictionary, the query performance of each determined database is calculated, and the database with the best query performance is selected as the database to be queried; Generate a data query task corresponding to the database to be queried according to the data table to be queried, the query dimension, and the query condition; Based on the preset read and write scheduling rules, the data query task is executed.
2. The method according to claim 1, characterized in that: The step of obtaining a data table to be processed according to the data query log stored in the heterogeneous database system includes: Query the data query log to obtain query information corresponding to the data table stored in the heterogeneous database system, the query information including: average query response time, query response monitoring index, query failure rate, and query frequency; If the query information corresponding to the data table meets the data synchronization condition, determining that the data table is the first data table that needs to be synchronized; If the query information corresponding to the data table meets the data migration condition, the data table is determined to be a second data table that needs data migration; wherein, The first data table and the second data table are the data tables to be processed.
3. The method according to claim 2, characterized in that The query information corresponding to the data table satisfies the data synchronization conditions, including at least one of the following options: the average query response time corresponding to the data table is greater than the preset average query response time, the query response index corresponding to the data table is greater than the preset query response index, the query failure rate corresponding to the data table is greater than the preset query failure rate, and the query frequency corresponding to the data table is greater than the first preset query frequency.
4. The method according to claim 2, characterized in that: The query information corresponding to the data table satisfies the data migration condition including: the query frequency corresponding to the data table is less than a second preset query frequency.
5. The method according to claim 2, characterized in that: The step of determining a target database corresponding to the to-be-processed data table includes: Based on the correspondence between the query information and the database type, determining a first target database for which data synchronization of the first data table is required according to the query information corresponding to the first data table; Based on the correspondence between database type and storage cost, and according to the storage information corresponding to the second data table, the second target database for data migration of the second data table and the migration data corresponding to the second data table are determined, wherein the storage information includes: storage database, storage time range, and storage cost.
6. The method according to claim 5, characterized in that The step of using the to-be-processed data table, the storage database and the target database to perform the data processing task includes: Synchronize the first data table from the first storage database corresponding to the first data table to the first target database according to the table structure corresponding to the first target database; According to the table structure corresponding to the second target database, the migration data corresponding to the second data table is migrated from the second storage database corresponding to the second data table to the second target database.
7. The method according to claim 1, characterized in that After executing the data processing task, the method further includes: Performing data verification on the to-be-processed data table and the target data table corresponding to the data processing task; If the verification fails, the data processing task is executed again.
8. The method according to claim 1, characterized in that The preset read / write scheduling rule includes at least one of the following options: If there is a data query task, execute the data query task first, then execute the data processing task; if there is no data query task, execute the data processing task directly; When the to-be-processed data table corresponding to the data processing task and the to-be-queried data table corresponding to the data query task are the same, the data processing task is executed first, and then the data query task is executed.
9. The method according to claim 1, characterized in that: The method further includes: generating a data query log corresponding to the data query request; and The data dictionary is updated using the generated data query log.
10. A data processing device based on a heterogeneous database system, characterized in that: include: A determination module, used to obtain a data table to be processed according to a data query log stored in a heterogeneous database system, and determine a target database corresponding to the data table to be processed; A generating module, used for generating a data processing task according to the to-be-processed data table, the storage database corresponding to the to-be-processed data table and the target database; An execution module, configured to execute the data processing task based on a preset read-write scheduling rule, using the data table to be processed, the storage database and the target database; wherein, after executing the data processing task, data verification is performed on the data table to be processed and the target data table corresponding to the data processing task; if the verification is passed, the storage information in the data dictionary is updated according to the data table to be processed and the target data table, and the query information in the data dictionary is updated using the data query log; A query module is used to receive a data query request and obtain input parameters corresponding to the data query request, wherein the input parameters include: a data table to be queried, a query dimension, and a query condition; Using the storage information and query information in the data dictionary, query the database to be queried corresponding to the data table to be queried; wherein, the database storing the data table to be queried is determined according to the data dictionary, the query performance of each determined database is calculated, and the database with the best query performance is selected as the database to be queried; Generate a data query task corresponding to the database to be queried according to the data table to be queried, the query dimension, and the query condition; Based on the preset read and write scheduling rules, the data query task is executed.
11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.
12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
