Data table mapping method, device, equipment and computer readable storage medium

Abstract

The present disclosure relates to a data table mapping method, device, equipment and computer readable storage medium. By responding to a target namespace configured by a user to be monitored, loading the target namespace to be monitored, monitoring whether each data table in the target namespace has a table creation action, in the case that the data table has a table creation action, creating a mapping base table based on a table creation statement in a pre-write log, the mapping base table including a primary key field and a default field corresponding to the data table. Compared with the prior art, the embodiment of the present disclosure monitors whether each data table in the target namespace has a table creation action, and in the case that the data table has a table creation action, creates a mapping base table based on a table creation statement in a pre-write log, can automatically perceive the creation and change of the data table of the database, can automatically adapt the mapping table for the database, can provide the ability to access the data table of the database in real time, and reduces the error of executing an SQL query through Phoenix.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data table mapping method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Phoenix is ​​a Java middleware that provides a Structured Query Language (SQL) layer for HBase databases, allowing developers to query data using standard SQL syntax. This makes it easier for developers familiar with SQL to use HBase. Phoenix operates on tables and data stored on HBase, achieving this by translating SQL queries into underlying operations that HBase can understand. Furthermore, Phoenix provides transaction support, which is crucial for applications that require ensuring data consistency.

[0003] To map data tables in HBase, Phoenix technology provides a dynamic column solution. This allows for the dynamic writing and querying of newly generated column fields when creating the Phoenix-to-HBase mapping table. However, it doesn't provide automatic tracking and mapping capabilities for HBase tables. On one hand, there are special scenarios where applications continuously create dynamic relational tables in HBase without knowing in advance which tables will be created, yet still want to use Phoenix's SQL capabilities after these tables are created. On the other hand, if the data table structure in HBase changes, and the manually created mapping table in Phoenix doesn't detect this change in time, errors may occur in Phoenix's SQL access.

[0004] However, the aforementioned dynamic column technology can only be effectively used if the user has a very good understanding of the relationship between HBase data tables and Phoenix mapping tables. It has poor flexibility and cannot automatically detect changes in HBase layer data tables in a timely manner, which may also lead to errors when executing SQL queries through Phoenix. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a data table mapping method, apparatus, device, and computer-readable storage medium that can automatically detect the creation and changes of data tables in a database, reducing errors when executing SQL queries through Phoenix.

[0006] In a first aspect, embodiments of this disclosure provide a data table mapping method, the method comprising:

[0007] In response to the user-configured target namespace to be monitored, load the target namespace to be monitored;

[0008] Monitor whether any table creation actions are being performed on any of the data tables in the target namespace;

[0009] If a table creation action is performed on the data table, a mapping base table is created based on the table creation statement in the write-ahead log. The mapping base table includes the primary key field and default field corresponding to the data table.

[0010] In some embodiments, before loading the target namespace to be monitored in response to a user-configured target namespace, the method further includes:

[0011] Start the mapping processor;

[0012] Based on the mapping processor, determine whether the database cluster is configured with a dynamic mapping table mechanism;

[0013] If the database cluster is not configured with a dynamic mapping table mechanism, then exit the mapping processor;

[0014] If the database cluster is configured with a dynamic mapping table mechanism, then the steps of loading the target namespace to be monitored are performed in response to the user-configured target namespace.

[0015] In some embodiments, monitoring whether there are table creation actions in each data table of the target namespace includes:

[0016] Read the write-ahead logs of the database cluster;

[0017] Based on the operation type in each log data entry of the write-ahead log, determine whether there is a table creation action in each data table of the target namespace.

[0018] In some embodiments, determining whether there are table creation actions for each data table in the target namespace based on the operation type in each log data entry of the write-ahead log includes:

[0019] For each log data entry in the write-ahead log, if the operation type in the log data is a target type, then it is determined that each data table in the target namespace has a table creation action.

[0020] In some embodiments, after creating the mapping base table based on the table creation statements in the write-ahead log, the method further includes:

[0021] Monitor changes to column fields in the database cluster;

[0022] The changed data based on the column fields is mapped to the mapping base table.

[0023] In some embodiments, monitoring changes to column fields in the database cluster includes:

[0024] Real-time monitoring of the database cluster's archive directory;

[0025] Determine if there are any newly transferred target write-ahead logs in the archive directory;

[0026] If there are newly transferred target write-ahead logs in the archive directory, the target write-ahead logs are moved to the parsing directory for parsing to obtain the parsed log data;

[0027] Find the target log data related to the mapping base table from the parsed log data;

[0028] Check whether the changed data in the column fields of the target log data has been mapped to the mapping base table;

[0029] If the changed data of the column field is not mapped to the mapping base table, then the changed data of the column field will be recorded in the change list.

[0030] In some embodiments, the change data based on the column field is mapped to the mapping base table, including:

[0031] Determine whether the target write-ahead log being parsed in the current batch meets the preset sampling rate;

[0032] If the preset sampling rate is met, the changes will be made to the mapping base table based on the changed data in the column fields of the change list;

[0033] If the preset sampling rate is not met, the parsed target write-ahead logs are cleaned up, and the process of monitoring changes in column fields in the database cluster continues.

[0034] Secondly, embodiments of this disclosure provide a data table mapping apparatus, the apparatus comprising:

[0035] The loading module is used to load the target namespace to be monitored in response to the user-configured target namespace;

[0036] The first monitoring module is used to monitor whether there are any table creation actions in each data table in the target namespace;

[0037] A creation module is used to create a mapping base table based on the table creation statements in the write-ahead log when there is a table creation action on the data table. The mapping base table includes the primary key field and default field corresponding to the data table.

[0038] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0039] Memory;

[0040] Processor; and

[0041] Computer programs;

[0042] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0043] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method as described in the first aspect.

[0044] Fifthly, embodiments of this disclosure also provide a computer program product comprising a computer program or instructions that, when executed by a processor, implement the method described in the first aspect.

[0045] The data table mapping method, apparatus, device, and computer-readable storage medium provided in this disclosure, in response to a user-configured target namespace to be monitored, load the target namespace to be monitored, monitor whether any table creation actions have occurred in each data table within the target namespace, and if table creation actions have occurred, create a mapping base table based on the table creation statements in the write-ahead log. The mapping base table includes the primary key field and default field corresponding to the data table. Compared to existing technologies, this disclosure monitors whether any table creation actions have occurred in each data table within the target namespace, and creates a mapping base table based on the table creation statements in the write-ahead log when table creation actions have occurred. This allows for automatic detection of database table creation and changes, automatic adaptation of the mapping table to the database, and provides users with more convenient real-time access to database tables using SQL. It also effectively controls the impact on database read / write performance and reduces errors when executing SQL queries through Phoenix. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0047] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart of a data table mapping method provided in this embodiment of the disclosure;

[0049] Figure 2 A flowchart illustrating the data table mapping to the underlying table provided in this embodiment of the disclosure;

[0050] Figure 3 A flowchart of a data table mapping method provided in another embodiment of this disclosure;

[0051] Figure 4 A flowchart of a data table mapping method provided in another embodiment of this disclosure;

[0052] Figure 5 A schematic diagram illustrating the process of dynamic column field data tracking provided in this embodiment of the disclosure;

[0053] Figure 6 This is a schematic diagram of the structure of the data table mapping device provided in the embodiments of this disclosure;

[0054] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0055] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0056] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0057] Phoenix is ​​a Java middleware that provides a Structured Query Language (SQL) layer for HBase databases, allowing developers to query data using standard SQL syntax. This makes it easier for developers familiar with SQL to use HBase. Phoenix operates on tables and data stored on HBase, achieving this by translating SQL queries into underlying operations that HBase can understand. Furthermore, Phoenix provides transaction support, which is crucial for applications that require ensuring data consistency.

[0058] To map data tables in HBase, Phoenix technology provides a dynamic column solution. This allows for the dynamic writing and querying of newly generated column fields when creating the Phoenix-to-HBase mapping table. However, it doesn't provide automatic tracking and mapping capabilities for HBase tables. On one hand, there are special scenarios where applications continuously create dynamic relational tables in HBase without knowing in advance which tables will be created, yet still want to use Phoenix's SQL capabilities after these tables are created. On the other hand, if the data table structure in HBase changes, and the manually created mapping table in Phoenix doesn't detect this change in time, errors may occur in Phoenix's SQL access.

[0059] However, the aforementioned dynamic column technology can only be effectively used if the user has a very good understanding of the relationship between HBase data tables and Phoenix mapping tables. It has poor flexibility and cannot automatically detect changes in HBase layer data tables in a timely manner, which may also lead to errors when executing SQL queries through Phoenix.

[0060] To address this issue, this disclosure provides a data table mapping method, which will be described below with reference to specific embodiments.

[0061] Figure 1 This is a flowchart illustrating the data table mapping method provided in this embodiment. The method is executed by an electronic device. The electronic device can be a portable mobile device such as a tablet or laptop; it can also be a fixed device such as a personal computer or a server. The server can be a single server, a server cluster, a distributed cluster, or a centralized cluster. This method can be applied to scenarios involving mapping data tables in a database. It is understood that the data table mapping method provided in this embodiment can also be applied to other scenarios.

[0062] The following is about Figure 1 The data table mapping method shown is introduced below. This method can be applied to electronic devices, and the specific steps included in the method are as follows:

[0063] S101. In response to the user-configured target namespace to be monitored, load the target namespace to be monitored.

[0064] The purpose of a namespace is to group data tables in HBase. HBase allows you to actively create a namespace and then create subsequent application data tables under this namespace. This isolates data tables for different business applications within different namespaces. HBase has two default namespaces: the HBase namespace and the default namespace. The HBase namespace primarily stores HBase system tables, while the default namespace is used for data tables that are not specified with a namespace.

[0065] In this step, the user configures the target namespace to be monitored. The electronic device responds to the user-configured target namespace by loading it. Only table creation actions within the target namespace will be monitored. Optionally, if the user does not customize the namespace to be monitored, the default namespace will be set as the target namespace.

[0066] S102. Monitor whether there are any table creation actions in each data table in the target namespace.

[0067] In this step, such as Figure 2 As shown, the electronic device monitors in real time whether any table creation actions are being performed on any of the data tables in the target namespace. The key to this monitoring lies in HBase's Write-Ahead Log (WAL), because all HBase data is written to the WAL. Therefore, changes to HBase data tables can be automatically detected by monitoring changes to the HBase WAL in real time.

[0068] S103. If there is a table creation action in the data table, create a mapping base table based on the table creation statement in the write-ahead log. The mapping base table includes the primary key field and the default field corresponding to the data table.

[0069] In this step, such as Figure 2 As shown, if table creation actions are detected in various data tables within the target namespace, the electronic device creates a mapping base table based on the table creation statements in the write-ahead log. This mapping base table includes the primary key field and default field corresponding to the data table. Optionally, the write-ahead log includes multiple log entries, each containing information such as the table name, the type of operation performed, column names, and time. The table name, operation type, column names, and time information in each log entry constitute a table creation statement.

[0070] The primary key field, also known as rowkey, corresponds to one rowkey for each data table in HBase, and all HBase data tables must have a rowkey; the default field is used to represent the column family field and serves as a placeholder field for the Phoenix mapping base table. Here, it only plays a placeholder role.

[0071] This embodiment of the disclosure, in response to a user-configured target namespace to be monitored, loads the target namespace and monitors whether any table creation actions have occurred in the target namespace. If a table creation action has occurred, a mapping base table is created based on the table creation statements in the write-ahead log. The mapping base table includes the primary key field and default field corresponding to the data table. Compared to the prior art, this embodiment of the disclosure monitors whether any table creation actions have occurred in the target namespace, and creates a mapping base table based on the table creation statements in the write-ahead log when a table creation action has occurred. This can automatically detect the creation and changes of database tables, automatically adapt the mapping table for the database, provide users with a more convenient ability to access database tables in real time using SQL, effectively control the impact on database data read and write performance, and reduce errors when executing SQL queries through Phoenix.

[0072] Figure 3 A flowchart of a data table mapping method provided in another embodiment of this disclosure is shown below. Figure 3 As shown, the method includes the following steps:

[0073] S201, Start the mapping processor.

[0074] like Figure 2 As shown, after the HBase cluster starts, electronic devices will automatically load Phoenix's mapping processor.

[0075] S202. Determine whether the database cluster is configured with a dynamic mapping table mechanism based on the mapping processor.

[0076] In this step, the electronic device checks whether a dynamic mapping table mechanism is configured for the database cluster through the mapping processor.

[0077] S203. If the database cluster is configured with a dynamic mapping table mechanism, then execute the step of loading the target namespace to be monitored in response to the user-configured target namespace.

[0078] In this step, such as Figure 2 As shown, when the database cluster starts the dynamic mapping table mechanism, the electronic device loads the target namespace to be monitored.

[0079] In some embodiments, if the database cluster is not configured with a dynamic mapping table mechanism, the mapping processor is exited.

[0080] like Figure 2 As shown, when the database cluster is not configured with a dynamic mapping table mechanism, the electronic device will exit the mapping processor.

[0081] S204. Read the write-ahead log of the database cluster.

[0082] In this step, the electronic device reads the Write-Ahead Log (WAL) of the database cluster. The WAL is a persistent log file used to record all data change operations in HBase. The purpose of the WAL is to ensure data durability and consistency. When data changes, the changes are first written to the WAL, and then to the data files in memory and on disk. This way, even in the event of a failure, data consistency can be restored through the WAL, ensuring no data loss. Furthermore, the WAL can also be used for data replication and recovery operations.

[0083] S205. Based on the operation type in each log data in the pre-written log, determine whether there is a table creation action in each data table in the target namespace.

[0084] In this step, the electronic device can determine whether there are table creation actions in each data table in the target namespace based on the operation type in each log data entry of the pre-written log, and can automatically detect the creation and changes of data tables in the database. Optionally, the operation type may include, but is not limited to, creation, deletion, and modification types.

[0085] In some embodiments, determining whether each data table in the target namespace has a table creation action based on the operation type in each log data in the write-ahead log includes: for each log data in the write-ahead log, if the operation type in the log data is the target type, then it is determined that each data table in the target namespace has a table creation action.

[0086] In this embodiment, the target type is a creation type. When the operation type in the log data is a creation type, the electronic device will determine that each data table in the target namespace has a table creation action.

[0087] S206. If there is a table creation action in the data table, create a mapping base table based on the table creation statement in the write-ahead log. The mapping base table includes the primary key field and the default field corresponding to the data table.

[0088] Specifically, the implementation process and principle of S206 and S103 are the same, and will not be repeated here.

[0089] S207. Monitor changes in column fields within the database cluster.

[0090] In this step, the electronic device initiates a dynamic data tracing process, monitoring changes in column fields within the database cluster in real time. Dynamic data tracing is primarily to accommodate the situation where HBase table columns change as data is written. This is because HBase does not specify column fields during table creation; actual column fields are only formed after data is continuously written. This process may result in newly added columns in the HBase table, therefore, we need to track the HBase data writing process to monitor all column changes.

[0091] S208. Based on the changed data of the column fields, map it to the mapping base table.

[0092] Furthermore, after detecting changes in column fields in the database cluster, the electronic device can map the changes in column fields to the mapping base table.

[0093] This embodiment of the disclosure starts a mapping processor, which determines whether the database cluster is configured with a dynamic mapping table mechanism. If the database cluster is configured with a dynamic mapping table mechanism, the step of loading the target namespace to be monitored, in response to the user-configured target namespace, is executed. Further, the write-ahead log of the database cluster is read, and based on the operation type in each log entry, it is determined whether there are table creation actions in each data table of the target namespace. If table creation actions are present, a mapping base table is created based on the table creation statements in the write-ahead log. The mapping base table includes the primary key field and default field corresponding to the data table. Next, changes in column fields in the database cluster are monitored. Based on these changes, the data is mapped to the mapping base table. This method can automatically detect the creation and changes of database tables, automatically adapt mapping tables for the database, provide users with a more convenient ability to access database tables in real time using SQL, effectively control the impact on database read / write performance, and reduce errors when executing SQL queries through Phoenix.

[0094] Figure 4 A flowchart of a data table mapping method provided in another embodiment of this disclosure is shown below. Figure 4 As shown, the method includes the following steps:

[0095] S301. In response to the user-configured target namespace to be monitored, load the target namespace to be monitored.

[0096] Specifically, the implementation process and principle of S301 and S101 are the same, and will not be repeated here.

[0097] S302. Monitor whether there are any table creation actions in each data table in the target namespace.

[0098] Specifically, the implementation process and principle of S302 and S102 are the same, and will not be repeated here.

[0099] S303. If there is a table creation action in the data table, create a mapping base table based on the table creation statement in the write-ahead log. The mapping base table includes the primary key field and the default field corresponding to the data table.

[0100] Specifically, the implementation process and principle of S303 and S103 are the same, and will not be repeated here.

[0101] S304, Real-time monitoring of the database cluster's archive directory.

[0102] In this step, such as Figure 5 As shown, the electronic device continuously monitors the database cluster's archive directory (oldWALs). Since all HBase data writes require the generation of WAL data files, the electronic device tracks column field changes by tracing the WAL files. However, to avoid the parsing process affecting HBase data write performance, the device continuously tracks the archive directory to monitor column field changes. The data in the archive directory has already been written to disk, so it does not affect real-time data writing.

[0103] S305. Determine whether there are any newly transferred target write-ahead logs in the archive directory.

[0104] In this step, such as Figure 5 As shown, the electronic device will determine whether there are any newly transferred target write-ahead logs in the archive directory. Optionally, the electronic device will periodically check the archive directory for newly transferred target write-ahead logs, for example, at a time of one minute, without limitation. The electronic device will check once every minute and determine whether there are any newly transferred target write-ahead logs within one minute based on the timestamp of the write-ahead logs. If so, S306 is executed; otherwise, the database cluster's archive directory is monitored.

[0105] S306. If there is a newly transferred target write-ahead log in the archive directory, the target write-ahead log is transferred to the parsing directory for parsing to obtain the parsed log data.

[0106] In this step, such as Figure 5 As shown, when a new target write-ahead log is transferred to the archive directory, the electronic device will move the newly transferred target write-ahead log to the parsing directory for parsing to obtain the parsed log data.

[0107] In S304 to S306, after the HBase cluster starts, it will continuously track the changes of WAL files in the oldWALs directory of HBase. When a new WAL file is found to be written to the directory, this part of the WAL will be moved to the parsing directory for parsing.

[0108] S307. Find the target log data related to the mapping base table from the parsed log data.

[0109] In this step, the electronic device will retrieve the target log data related to the mapping base table from the parsed log data. Specifically, such as... Figure 5 As shown, it is determined whether the parsed log data is the log data of the mapping base table that needs to be tracked, that is, whether the parsed log data is related to the mapping base table. If so, the change data of the column fields in the target log data is obtained.

[0110] S308. Check whether the changed data of the column fields in the target log data has been mapped to the mapping base table.

[0111] In this step, the electronic device obtains the change data of the column fields in the target log data, determines whether the change data of the column fields in the target log data has been mapped to the mapping base table, and if so, obtains the change data of the next column field and determines whether the change data of the next column field has been mapped to the mapping base table; if not, it executes S309.

[0112] S309. If the changed data of the column field is not mapped to the mapping base table, then the changed data of the column field is recorded in the change list.

[0113] like Figure 5 As shown, if it is determined that the changed data of the column field is not mapped to the mapping base table, the electronic device will record the changed data of the column field in the change list.

[0114] In S307 to S309, after parsing the newly transferred WAL file in the current batch, the relevant data of the mapping base table that needs to be continuously tracked will be searched. The column fields in the data will be compared to see if they have been recorded in the corresponding Phoenix mapping base table. If the mapping has been completed, the data will be filtered out; otherwise, the column fields will be recorded in the change list.

[0115] S310. Based on the changed data of the column fields, map it to the mapping base table.

[0116] Specifically, the implementation process and principle of S310 and S208 are the same, and will not be repeated here.

[0117] In some embodiments, S310 may include, but is not limited to, S3101, S3102, and S3103:

[0118] S3101. Determine whether the target write-ahead log parsed in the current batch meets the preset sampling rate.

[0119] In this embodiment, the electronic device determines whether the target write-ahead log parsed in the current batch meets the preset sampling rate.

[0120] A preset sampling rate is introduced to avoid excessive parsing time and resources consumption in the HBase cluster due to a large volume of WAL data. It also considers that HBase table column changes are typically infrequent. Cluster administrators can set a sampling rate between 0.01 and 1 based on HBase cluster business changes. This configuration will then extract a specified percentage of data from the current batch of WAL files for parsing. A setting of 1 means all WAL data needs parsing, offering the highest accuracy and real-time performance, but also consuming the most cluster resources and taking the longest time. A setting of 0.01 parses only one percent of the data, effectively reducing resource consumption and parsing time, but offering the lowest accuracy and real-time performance, potentially resulting in incomplete parsing of the current batch and the omission of newly added columns. The default setting is usually 1.

[0121] S3102. If the preset sampling rate is met, the changed data of the column fields in the change list are changed to the mapping base table.

[0122] like Figure 5 As shown, if the target write-ahead log parsed in the current batch meets the preset sampling rate, the electronic device will map the changed data of the column fields in the change list, i.e., the newly added column fields, to the corresponding mapping base table of Phoenix.

[0123] S3103. If the preset sampling rate is not met, clean up the parsed target write-ahead log and continue to execute the steps of monitoring the change data of column fields in the database cluster.

[0124] like Figure 5 As shown, if the target write-ahead log parsed in the current batch does not meet the preset sampling rate, the electronic device will clean up the parsed target write-ahead log and continue to execute the step of monitoring the change data of the column fields in the database cluster until the preset sampling rate is met. Then, based on the change data of the column fields in the change list, the data will be changed to the mapping base table.

[0125] This embodiment of the disclosure loads the target namespace to be monitored in response to a user-configured target namespace, monitors whether any table creation actions occur in the data tables within the target namespace, and creates a mapping base table based on the table creation statements in the write-ahead log when table creation actions occur. The mapping base table includes the primary key field and default field corresponding to the data table. Further, the archive directory of the database cluster is monitored in real time to determine if any newly transferred target write-ahead logs are present in the archive directory. If so, the target write-ahead logs are moved to a parsing directory for parsing to obtain the parsed log data. Target log data related to the mapping base table is then retrieved from the parsed log data. Next, it is checked whether the changed data of the column fields in the target log data has been mapped to the mapping base table. If the changed data of the column fields has not been mapped to the mapping base table, the changed data of the column fields is recorded in a change list. Finally, based on the changed data of the column fields, the data is mapped to the mapping base table. Compared to existing technologies, the embodiments of this disclosure can automatically detect the creation and changes of database tables, automatically adapt mapping tables for the database, provide users with a more convenient ability to access database tables in real time using SQL, effectively control the impact on database data read and write performance, and reduce errors when executing SQL queries through Phoenix.

[0126] Figure 6 This is a schematic diagram of the structure of a data table mapping device provided in an embodiment of this disclosure. The data table mapping device can be an electronic device as described in the above embodiments, or it can be a component or assembly within that electronic device. The data table mapping device provided in this embodiment can execute the processing flow provided in the data table mapping method embodiments, such as... Figure 6 As shown, the data table mapping device 50 includes: a loading module 51, a first listening module 52, and a creation module 53; wherein, the loading module 51 is used to load the target namespace to be listened to in response to the target namespace configured by the user; the first listening module 52 is used to listen for whether there is a table creation action in each data table in the target namespace; the creation module 53 is used to create a mapping base table based on the table creation statement in the write-ahead log when the data table has a table creation action, and the mapping base table includes the primary key field and default field corresponding to the data table.

[0127] Optionally, before loading the target namespace to be monitored in response to the user-configured target namespace, the data table mapping device 50 further includes: a judgment module 54; the judgment module 54 is used to start the mapping processor; determine whether the database cluster is configured with a dynamic mapping table mechanism based on the mapping processor; if the database cluster is not configured with a dynamic mapping table mechanism, then exit the mapping processor; if the database cluster is configured with a dynamic mapping table mechanism, then execute the step of loading the target namespace to be monitored in response to the user-configured target namespace.

[0128] Optionally, when the first monitoring module 52 monitors whether there are table creation actions in each data table in the target namespace, it is specifically used to: read the write-ahead log of the database cluster; and determine whether there are table creation actions in each data table in the target namespace based on the operation type in each log data in the write-ahead log.

[0129] Optionally, when the first monitoring module 52 determines whether there is a table creation action in each data table in the target namespace based on the operation type in each log data in the write-ahead log, it is specifically used to: for each log data in the write-ahead log, if the operation type in the log data is the target type, then it is determined that there is a table creation action in each data table in the target namespace.

[0130] Optionally, after creating the mapping base table based on the table creation statement in the write-ahead log, the data table mapping device 50 further includes: a second listening module 55 and a mapping module 56; the second listening module 55 is used to listen for changes in column fields in the database cluster; the mapping module 56 is used to map the changes in the column fields to the mapping base table.

[0131] Optionally, when the second monitoring module 55 monitors changes in column fields in the database cluster, it is specifically used for: real-time monitoring of the database cluster's archive directory; determining whether there are newly transferred target write-ahead logs in the archive directory; if there are newly transferred target write-ahead logs in the archive directory, transferring the target write-ahead logs to the parsing directory for parsing to obtain parsed log data; finding target log data related to the mapping base table from the parsed log data; checking whether the changed data of the column fields in the target log data has been mapped to the mapping base table; if the changed data of the column fields has not been mapped to the mapping base table, recording the changed data of the column fields in the change list.

[0132] Optionally, when the mapping module 56 maps the changed data of the column fields to the mapping base table, it is specifically used to: determine whether the target write-ahead log parsed in the current batch meets the preset sampling rate; if the preset sampling rate is met, then change the changed data of the column fields in the change list to the mapping base table; if the preset sampling rate is not met, then clean up the parsed target write-ahead log and continue to execute the step of monitoring the changed data of the column fields in the database cluster.

[0133] Figure 6 The data table mapping device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0134] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 7 It shows a schematic diagram of a structure suitable for implementing the electronic device 600 in the embodiments of this disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0135] like Figure 7 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603 to implement the data table mapping method as described in the embodiments of this disclosure. Various programs and data required for the operation of electronic device 600 are also stored in RAM 603. The processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0136] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0137] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the data table mapping method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 609, or installed from storage device 608, or installed from ROM 602. When the computer program is executed by processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0138] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0139] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0140] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0141] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0142] In response to the user-configured target namespace to be monitored, load the target namespace to be monitored;

[0143] Monitor whether any table creation actions are being performed on any of the data tables in the target namespace;

[0144] If a table creation action is performed on the data table, a mapping base table is created based on the table creation statement in the write-ahead log. The mapping base table includes the primary key field and default field corresponding to the data table.

[0145] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0146] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0149] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0150] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0152] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0153] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data table mapping method, characterized in that, The method includes: In response to the user-configured target namespace to be monitored, load the target namespace to be monitored; Monitor whether any table creation actions are being performed on any of the data tables in the target namespace; If a table creation action is performed on the data table, a mapping base table is created based on the table creation statement in the write-ahead log. The mapping base table includes the primary key field and default field corresponding to the data table.

2. The method according to claim 1, characterized in that, Before loading the target namespace to be monitored in response to the user-configured target namespace, the method further includes: Start the mapping processor; Based on the mapping processor, determine whether the database cluster is configured with a dynamic mapping table mechanism; If the database cluster is not configured with a dynamic mapping table mechanism, then exit the mapping processor; If the database cluster is configured with a dynamic mapping table mechanism, then the steps of loading the target namespace to be monitored are performed in response to the user-configured target namespace.

3. The method according to claim 1, characterized in that, The monitoring of whether any tables in the target namespace have been created includes: Read the write-ahead logs of the database cluster; Based on the operation type in each log data entry of the write-ahead log, determine whether there is a table creation action in each data table of the target namespace.

4. The method according to claim 1, characterized in that, The step of determining whether there are table creation actions in each data table of the target namespace based on the operation type in each log data in the write-ahead log includes: For each log data entry in the write-ahead log, if the operation type in the log data is a target type, then it is determined that each data table in the target namespace has a table creation action.

5. The method according to claim 1, characterized in that, After creating the mapping base table based on the table creation statements in the write-ahead log, the method further includes: Monitor changes to column fields in the database cluster; The changed data based on the column fields is mapped to the mapping base table.

6. The method according to claim 5, characterized in that, The monitoring of changes to column fields in the database cluster includes: Real-time monitoring of the database cluster's archive directory; Determine if there are any newly transferred target write-ahead logs in the archive directory; If there are newly transferred target write-ahead logs in the archive directory, the target write-ahead logs are moved to the parsing directory for parsing to obtain the parsed log data; Find the target log data related to the mapping base table from the parsed log data; Check whether the changed data in the column fields of the target log data has been mapped to the mapping base table; If the changed data of the column field is not mapped to the mapping base table, then the changed data of the column field will be recorded in the change list.

7. The method according to claim 5, characterized in that, The changed data based on the column fields is mapped to the mapping base table, including: Determine whether the target write-ahead log being parsed in the current batch meets the preset sampling rate; If the preset sampling rate is met, the changes will be made to the mapping base table based on the changed data in the column fields of the change list; If the preset sampling rate is not met, the parsed target write-ahead logs are cleaned up, and the process of monitoring changes in column fields in the database cluster continues.

8. A data table mapping device, characterized in that, The device includes: The loading module is used to load the target namespace to be monitored in response to the user-configured target namespace; The first monitoring module is used to monitor whether there are any table creation actions in each data table in the target namespace; A creation module is used to create a mapping base table based on the table creation statements in the write-ahead log when there is a table creation action on the data table. The mapping base table includes the primary key field and default field corresponding to the data table.

9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images