Database management method, system, electronic device and computer readable storage medium

Abstract

The application discloses a database management method, system, electronic equipment and computer readable storage medium. The method comprises the following steps: in response to the metadata of the database changing, the control module of the database matching perceives that the metadata of the database changes, and the data center receives the metadata message corresponding to the metadata generated by the control module; wherein the metadata of the database is related to the data table in the database, and the metadata message is generated based on the structure information of the data table in the database; a synchronization task corresponding to the metadata message is generated, the temporary data table in the database is determined by executing the synchronization task, and the structure information corresponding to the temporary data table is updated to the storage module of the data center; wherein the temporary data table is different from the pre-defined data table defined by the data center; and the temporary data table in the database is managed based on the structure information stored in the storage module. The above scheme can improve the convenience of database management and increase the available storage capacity of the database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and particularly relates to a database management method and system, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the advent of the data era, massive data needs to rely on a database to accommodate and manage, therefore, the management of the database is increasingly valued. A large number of data tables are usually created in the database, and the data tables occupy the available storage of the database. In the prior art, the changes of the data tables in the database are difficult to obtain in time, so that the data tables are continuously accumulated to reduce the available storage of the database. When it is necessary to release the space of the database, the data tables need to be managed in an intrusive manner, so that the convenience of managing the database is not high. Therefore, how to improve the convenience of managing the database and increase the available storage of the database becomes a problem to be solved. SUMMARY

[0003] The technical problem solved by the present application is to provide a database management method and system, an electronic device and a computer readable storage medium, which can improve the convenience of managing the database and increase the available storage of the database.

[0004] To solve the above technical problem, the first aspect of the present application provides a database management method, comprising: in response to the metadata of the database changing being perceived by a control module matched with the database, a data center receiving a metadata message corresponding to the metadata generated by the control module; wherein the metadata of the database is related to a data table in the database, the metadata message is generated based on structure information of the data table in the database, and the structure information at least includes a database name corresponding to the data table, table information and a change type; a synchronization task corresponding to the metadata message is generated, a temporary data table in the database is determined by executing the synchronization task, and structure information corresponding to the temporary data table is updated to a storage module of the data center; wherein the temporary data table is different from a predefined data table defined by the data center; and based on the structure information stored in the storage module, the temporary data table in the database is managed in the data center.

[0005] To solve the above technical problems, the second aspect of the present application provides a database management system, comprising: a control module and a data center, the control module is used for sensing the change of metadata of the database, and generating a metadata message corresponding to the metadata; wherein the metadata of the database is related to a data table in the database, the metadata message is generated based on structure information of the data table in the database, and the structure information at least includes a database name corresponding to the data table, table information and a change type; the data center is used for receiving the metadata message generated by the control module; generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating structure information corresponding to the temporary data table to a storage module; wherein the data center comprises the storage module, the temporary data table is different from a predefined data table defined by the data center, and the temporary data table in the database can be managed based on the structure information stored in the storage module.

[0006] To solve the above technical problems, the third aspect of the present application provides an electronic device, comprising: a storage and a processor coupled with each other, wherein the storage stores program data, and the processor invokes the program data to execute the method of the first aspect.

[0007] To solve the above technical problems, the fourth aspect of the present application provides a computer storage medium, which stores program data, and the program data is executed by a processor to implement the method of the first aspect.

[0008] The above scheme, when the control module of the database matches the database and senses that the metadata of the database changes, the data center receives the metadata message corresponding to the metadata generated by the control module, the metadata in the database is related to the data table in the database, and the metadata message is generated based on the structure information of the data table in the database, that is, when the data table in the database changes, the metadata of the database is sensed by the control module, the control module generates the metadata message based on the structure information of the data table to make the data center receive, wherein the structure information at least includes the database name corresponding to the data table, the table information and the change type, when the data center obtains the metadata message, generates a synchronization task corresponding to the metadata message, so that the data center executes the synchronization task to determine the temporary data table in the database which is different from the predefined data table, and updates the structure information corresponding to the temporary data table in the storage module of the data center. Since the structure information includes the database name corresponding to the data table, the table information and the change type, the temporary data table in the database can be managed in the data center based on the structure information stored in the storage module, so as to release the storage space occupied by the temporary data table, improve the convenience of database management, and increase the available storage amount of the database. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and other drawings can be obtained by those of ordinary skill in the art without any creative effort based on these drawings. Among them:

[0010] Figure 1 is a flow diagram of an embodiment of the database management method of the present application;

[0011] Figure 2 is a flow diagram of another embodiment of the database management method of the present application;

[0012] Figure 3 is a structural diagram of an embodiment of the database management system of the present application;

[0013] Figure 4 is a topological structure diagram of an embodiment of the database management system of the present application;

[0014] Figure 5 is a structural diagram of an embodiment of the electronic device of the present application;

[0015] Figure 6 is a structural diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort belong to the scope of protection of the present application.

[0017] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" in this document is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " in this document generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" in this document means two or more than two.

[0018] The database management method provided by the application is used for managing a database, for example, a Hive database, and the database management method corresponds to a database management system to implement the above-mentioned database management method, wherein the database management system at least includes a control module matched with the database and a data center independent of the control module, and the database management method provided by the application takes the data center as an execution subject.

[0019] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the database management method of the application, and the method includes:

[0020] S101: In response to the database matching control module perceiving that the metadata of the database changes, the data center receives the metadata message corresponding to the metadata generated by the control module, wherein the metadata of the database is related to the data table in the database, the metadata message is generated based on the structure information of the data table in the database, and the structure information at least includes the database name corresponding to the data table, table information and change type.

[0021] Specifically, when the database matching control module perceives that the metadata of the database changes, the data center receives the metadata message corresponding to the metadata generated by the control module, the metadata in the database is related to the data table in the database, and the metadata message is generated based on the structure information of the data table in the database, wherein the structure information at least includes the database name corresponding to the data table, table information and change type.

[0022] Further, when the data table in the database changes, the metadata of the database is perceived by the control module after the change, the control module generates the metadata message based on the structure information of the data table to enable the data center to receive, and the data center receives the metadata message generated by the control module, so that the data center can conveniently obtain the metadata message in the database.

[0023] In an application mode, the control module corresponds to multiple databases, when the data table in any one or more databases changes, the metadata of the database changes due to the change of the data table, wherein the change type of the data table includes but is not limited to adding a table, editing a table and deleting a table, and then the database matching control module perceives that the metadata of the database changes, and generates the metadata message based on the structure information corresponding to the changed data table, and the data center receives the metadata message, wherein the structure information includes the database name, table information and change type.

[0024] In another application mode, the number of control modules corresponds to the number of databases, each control module corresponds to a matched database, when the data table in any database changes, the metadata of the corresponding database changes due to the change of the data table, wherein the change type of the data table includes but is not limited to adding a table, editing a table and deleting a table, and then the control module matched with the database where the data table changes perceives that the metadata of the database changes, and generates a metadata message based on the structure information of the changed data table, and the data center receives the metadata message, wherein the structure information includes database name, table information and change type.

[0025] In an application scenario, the database is a Hive database, and the control module and the data center transmit metadata messages through a Kafka message queue, when a new data table is created in the database or a data table is edited or deleted, the data table in the database changes and is perceived by the control module, the control module generates a metadata message based on the structure information of the changed data table, and the data center receives the metadata message sent by the control module in the form of a consumer.

[0026] S102: Generate a synchronization task corresponding to the metadata message, execute the synchronization task to determine the temporary data table in the database, and update the structure information of the temporary data table to the storage module of the data center, wherein the temporary data table is different from the pre-defined data table defined by the data center.

[0027] Specifically, after the data center obtains the metadata message, a synchronization task corresponding to the metadata message is generated, so that the data center executes the synchronization task to determine the temporary data table in the database that is different from the pre-defined data table, and the structure information of the temporary data table is updated in the storage module of the data center, so that the data center can update the structure information of the temporary data table in the database in a timely manner, so as to facilitate the management of the temporary data table.

[0028] It should be noted that the pre-defined data table is pre-set, and the pre-defined data table is a data table that has been included in the management of the data center. After filtering the pre-defined data table, the structure information of the pre-defined data table is not included in the storage module, thereby effectively protecting the data security of the pre-defined data table.

[0029] In an application mode, the data center generates a synchronization task based on the metadata message, sends the synchronization task to an idle thread for execution, thereby determining the data table corresponding to the metadata message by parsing the metadata message, and comparing with the pre-defined data table, taking the data table in the database that is different from the pre-defined data table as a temporary data table, and updating the structure information of the parsed temporary data table in the storage module of the data center.

[0030] In another application mode, the metadata message includes sub-messages corresponding to the data table, the data center obtains the sub-messages in the metadata message after obtaining the metadata message, generates sub-tasks corresponding to the sub-messages based on the sub-messages, wherein the synchronization task includes all the sub-tasks, and the all sub-tasks are batched and sent to idle threads for execution, so as to parse the metadata message to determine the data table corresponding to the metadata message, and compare with the predefined data table, and the data table different from the predefined data table in the database is taken as a temporary data table, and the structure information of the parsed temporary data table is updated in the storage module of the data center.

[0031] S103: Based on the structure information stored in the storage module, the temporary data table in the database is managed in the data center.

[0032] Specifically, the data center can call the structure information of the temporary data table from the storage module, so as to consult the temporary data table in the database in the data center, and delete the temporary data table in the database based on the structure information, so as to manage the data table in the database in the data center, thereby effectively releasing the storage space occupied by the temporary data table, improving the convenience of database management, and increasing the available storage of the database.

[0033] In an application mode, based on the structure information stored in the data center, the data amount of the temporary data table is determined, and when the data amount of the temporary data table exceeds the data amount threshold, at least part of the temporary data table in the database is deleted to release the storage space occupied by at least part of the temporary data table, thereby increasing the available storage of the database.

[0034] In another application mode, based on the structure information stored in the storage module, the data of the temporary data table and / or the data amount of the temporary data table are viewed in the data center; based on the structure information stored in the storage module, a deletion instruction for deleting at least part of the temporary data table is sent to the database in the data center. Wherein, the data center supports deleting the temporary data table, previewing the data in the temporary data table, and querying the storage amount of the data in the temporary data table, the user can view the data of the temporary data table, or view the data amount of the temporary data table, or view the data and the data amount of the temporary data table in the data center, and then obtain the selection result of the user, and based on the structure information stored in the storage module, a deletion instruction for deleting the temporary data table specified by the user is sent to the database in the data center, so as to provide customized services for the user in the data center, and improve the convenience of database management.

[0035] In an application scenario, the functions supported by the data middle platform include, but are not limited to, viewing a temporary data table, deleting the temporary data table, previewing data in the temporary data table, querying a storage amount of data in the temporary data table, and querying modification records of the temporary data table. Thus, a user can view and delete the temporary data table based on the above functions, so as to manage the temporary data table in the database.

[0036] The above scheme, when the database matching control module perceives that the metadata of the database changes, the data middle platform receives the metadata message corresponding to the metadata generated by the control module, the metadata in the database is related to the data table in the database, and the metadata message is generated based on the structure information of the data table in the database, that is, when the data table in the database changes, the metadata of the database is perceived after the change and is generated by the control module based on the structure information of the data table to make the data middle platform receive, wherein the structure information at least includes the database name, table information and change type corresponding to the data table, when the data middle platform obtains the metadata message, generates a synchronization task corresponding to the metadata message, so that the data middle platform executes the synchronization task to determine the temporary data table in the database that is different from the pre-defined data table, and updates the structure information corresponding to the temporary data table in the storage module of the data middle platform. Since the structure information includes the database name, table information and change type corresponding to the data table, the temporary data table in the database can be managed in the data middle platform based on the structure information stored in the storage module, so as to release the storage space occupied by the temporary data table and improve the convenience of database management and increase the available storage amount of the database.

[0037] Please refer to Figure 2 , Figure 2 is a flowchart of another embodiment of the database management method of the present application, which comprises:

[0038] S201: In response to the database matching control module perceiving that the metadata of the database changes, the data middle platform receives the metadata message generated by the control module based on a preset protocol, wherein the metadata of the database is related to the data table in the database, and the metadata message is generated based on the structure information of the data table in the database, and the structure information at least includes the database name, table information and change type corresponding to the data table.

[0039] Specifically, the data middle platform is connected to at least one database, and the control module monitors the metadata of the database in real time. When the data table in any database changes, the control module generates a metadata message based on the preset protocol and the structure information corresponding to the changed data table. Each database with changed metadata has its own metadata message.

[0040] It should be noted that the database-based type customization has a control module, the control module monitors the metadata of the database in real time, thereby improving the efficiency of obtaining the metadata message, when the data table in any database changes, the control module encodes the structure information corresponding to the changed data table based on the preset protocol, generates a metadata message, when the data center corresponds to multiple databases and the metadata of the multiple databases changes, each database with changed metadata corresponds to its own metadata message, so that the metadata messages corresponding to different databases can be executed by different threads, thereby improving the execution efficiency and reducing the probability of metadata message disorder between databases.

[0041] Further, after the control module generates the metadata message, the data center receives the metadata message generated by the control module based on the preset protocol.

[0042] In an application scenario, the control module and the data center transmit the metadata message based on the Kafka message queue, and the metadata message includes at least one sub-message corresponding to at least one data table in the database, the sub-message is related to the change type, and the change type includes creating a table, editing a table, and deleting a table.

[0043] Further, receiving the metadata message generated by the control module based on the preset protocol includes: calling an interface of the control module to obtain a cluster address of the message queue; obtaining multiple sub-messages in the metadata message based on the cluster address, and determining the position index of the sub-message in the message queue.

[0044] Specifically, the control module and the data center use the Kafka message queue to efficiently and low-latency transmit the metadata message, the metadata message includes at least one sub-message corresponding to at least one data table in the database, the sub-message is related to the change type, for example, a sub-message related to creating a table, a sub-message related to editing a table, or a sub-message related to deleting a table, when the data table corresponds to multiple sub-messages or multiple data tables correspond to sub-messages, the sub-messages are arranged in the message queue in sequence and received by the data center.

[0045] Further, the data center calls an interface of the control module to obtain a cluster address of the message queue, if the call fails, retries after a preset time interval until the cluster address is successfully obtained, obtains multiple sub-messages in the metadata message based on the cluster address, and determines the position index of the sub-message in the message queue, so that when the synchronization task is interrupted, the position index can be used to continue the synchronization task, thereby improving the execution efficiency of the synchronization task, wherein the preset time interval can be 20 seconds or other time intervals.

[0046] In a specific application scenario, the structure information contained in the sub-message related to creating a table includes a database name, a table creation time, a type of the time, metadata object information, table information, a field list, a partition information list, a storage format, a compression mode, a table note, and a table name; the structure information contained in the sub-message related to editing a table includes a database name, a table editing time, a type of the time, metadata object information, table information, a field list, a partition information list, a storage format, a compression mode, a table note, a table name, old table information, and an old table name; and the structure information contained in the sub-message related to deleting a table includes a database name, a table deletion time, a type of the time, metadata object information, table information, and old table information.

[0047] S202: A synchronization task corresponding to the metadata message is generated, the synchronization task is executed to determine a temporary data table in the database, and structure information of the temporary data table is updated to a storage module of the data center.

[0048] Specifically, the control module and the data center transmit the metadata message based on the Kafka message queue, and the metadata message includes at least one sub-message corresponding to at least one data table in the database. The data center corresponds to a predefined data table. After the data center obtains the metadata message, it generates a synchronization task based on all sub-messages in the metadata message. When the synchronization task is executed, the data tables in the database are compared with the predefined data table to determine a temporary data table in the database. The structure information of the temporary data table is updated to the storage module of the data center.

[0049] In an application mode, a synchronization sub-task corresponding to the metadata message is generated based on the number of sub-messages in the metadata message, and the synchronization sub-task is sent to an idle thread for execution. The number of sub-messages in each synchronization sub-task does not exceed a number threshold. Based on the data table corresponding to the sub-message in the synchronization sub-task and the predefined data table, a temporary data table in the database is determined, and the structure information of the temporary data table is updated in the storage module of the data center based on the sub-message corresponding to the temporary data table.

[0050] Specifically, the number of sub-messages in the metadata message is batch-pulled based on the number of sub-messages in the metadata message, and the number of sub-messages pulled each time does not exceed a number threshold. A synchronization sub-task is generated based on the pulled sub-messages, and the synchronization sub-task is sent to an idle thread for execution. Thus, the metadata message corresponding to the data table in the database is synchronized to the data center. In the synchronization process, the data table corresponding to the sub-message in the synchronization sub-task is compared with the predefined data table of the data center to determine a temporary data table in the database that is different from the predefined data table. Then, the structure information of the temporary data table is updated in the storage module of the data center based on the sub-message corresponding to the temporary data table, so as to improve the accuracy of the structure information in the data center.

[0051] In an application scenario, based on the data table corresponding to the sub-message in the synchronization sub-task and the predefined data table, the temporary data table in the database is determined, and based on the sub-message corresponding to the temporary data table, the structure information of the temporary data table is updated in the storage module of the data platform, including: based on the predefined data table, filtering the data table in the sub-message corresponding to the data table that is repeated with the predefined data table, obtaining the temporary data table in the database; traversing all sub-messages corresponding to the temporary data table, based on the change type corresponding to the sub-message, calling the update mode corresponding to the change type, and updating the structure information of the temporary data table to the storage module.

[0052] Specifically, different change types correspond to respective update modes, after obtaining the synchronization sub-task, the data table corresponding to the sub-message in the synchronization sub-task is compared with the predefined data table, the data table in the sub-message corresponding to the data table that is repeated with the predefined data table is filtered, the temporary data table in the database is obtained, for each sub-message corresponding to the temporary data table, based on the change type corresponding to the sub-message, the update mode corresponding to the change type is called, the structure information corresponding to the sub-message is parsed, and the structure information of the temporary data table is updated to the storage module according to the update mode matched with the change type.

[0053] In a specific application scenario, the number threshold is 200, when the number of sub-messages in the metadata message exceeds the number threshold, 200 sub-messages are pulled at a time, a synchronization sub-task is generated, the synchronization sub-task is sent to an idle thread for execution, the predefined data table coinciding with the data platform is filtered to determine the temporary data table in the database, based on the message type corresponding to the sub-message of the temporary data table, when the message type is create table, it is determined that the current temporary data table is not in the storage module, then the update method matched with the create table is called, the structure information corresponding to the temporary data table is updated to the storage module of the data platform, when the message type is edit table, it is determined that the current temporary data table is in the storage module, then the update method matched with the edit table is called, the structure information corresponding to the temporary data table is updated to the storage module of the data platform, when the message type is delete table, it is determined that the current temporary data table is in the storage module, then the update method matched with the delete table is called, and the structure information corresponding to the temporary data table is deleted from the storage module of the data platform.

[0054] S203: In response to the management service module publishing an event for the database, the data service module monitors the event, and determines the task state of the synchronization task corresponding to the metadata message based on the event.

[0055] Specifically, the data middle platform further comprises a management service module and a data service module, the event comprises binding a database, editing a database and unbinding a database, and the metadata of the database corresponding to the event changes after the event is executed. The event comprises an operation on the database, the binding of a new database corresponds to the binding of the database, the editing of the database corresponds to the modification of a username and / or login password of the database, and the unbinding of the database corresponds to the unbinding of the bound database. After the event is executed, the metadata of the database changes.

[0056] Further, when the management service module publishes an event for the database, the data service module monitors the event, that is, when the data service module monitors the event, the database changes, and then the task state of the synchronization task corresponding to the metadata message needs to be determined based on the event, wherein the task state comprises starting a new synchronization task, pausing the synchronization task and terminating the synchronization task, so as to confirm the task state in time when the database changes, so as to determine how to execute the synchronization task.

[0057] In an application scenario, the task state of the synchronization task corresponding to the metadata message is determined based on the event, comprising: in response to the event being the binding of a database, starting to receive the metadata message corresponding to the newly bound database generated by the control module from the beginning, and starting the synchronization task corresponding to the newly bound database; in response to the event being the editing of a database, pausing the synchronization task corresponding to the edited database, and recording the position index of the synchronized sub-message in the message queue; and in response to the event being the unbinding of a database, terminating the synchronization task corresponding to the unbound database.

[0058] Specifically, when the event is the binding of a database, the metadata message corresponding to the newly bound database generated by the control module is received from the beginning, and the synchronization task corresponding to the newly bound database is started. When the event is the editing of a database, the synchronization task corresponding to the edited database is paused, and the position index of the synchronized sub-message in the message queue is recorded. When the event is the unbinding of a database, the synchronization task corresponding to the unbound database is terminated. The synchronization task corresponding to the newly bound database is in a starting state, the synchronization task corresponding to the edited database is in a paused state, and the synchronization task corresponding to the unbound database is in a terminated state. The metadata message of the database is in a continuous updating process, and the structure information corresponding to the data table in the unbound database is deleted from the storage module, thereby improving the execution efficiency and rationality of the synchronization task.

[0059] S204: Based on the structure information stored in the storage module, the temporary data table in the database is managed in the data middle platform.

[0060] Specifically, the data hub can call the structure information of the temporary data table from the storage module, and manage the temporary data table based on the structure information stored in the storage module.

[0061] In an application scenario, based on the structure information stored in the storage module, the data hub views the data of the temporary data table and / or the data volume of the temporary data table; based on the structure information stored in the storage module, the data hub sends a deletion instruction to delete at least part of the temporary data table to the database.

[0062] Specifically, the data hub supports deleting the temporary data table, previewing the data in the temporary data table, and querying the storage volume of the data in the temporary data table. The user can view the data of the temporary data table, or view the data volume of the temporary data table, or view the data and the data volume of the temporary data table in the data hub, and then obtain the selection result of the user. Based on the structure information stored in the storage module, the data hub sends a deletion instruction to delete the temporary data table specified by the user to the database, so as to provide customized services for the user in the data hub and improve the convenience of database management.

[0063] In the embodiment, the control module and the data hub use the Kafka message queue to efficiently and low-latency transmit metadata messages. The metadata messages include at least one sub-message related to the change type corresponding to at least one data table in the database. Based on the number of sub-messages in the metadata messages, the sub-messages in the metadata messages are batch-pulled, and the number of sub-messages pulled each time does not exceed a number threshold. Based on the pulled sub-messages, a synchronization sub-task is generated. For the data table and the predefined data table of the data hub corresponding to the sub-messages in the synchronization sub-task, the temporary data table different from the predefined data table in the database is determined. Then, based on the sub-messages corresponding to the temporary data table, the structure information of the temporary data table is updated in the storage module of the data hub, so as to improve the accuracy of the structure information in the data hub. When the management service module publishes an event for the database, the data service module monitors the event. Based on the event, the task state of the synchronization task corresponding to the metadata message is determined, so as to improve the efficiency and rationality of the execution of the synchronization task. When the storage module of the data hub corresponds to the structure information of the temporary data table, the temporary data table in the database can be managed in the data hub, so as to release the storage space occupied by the temporary data table and improve the convenience of database management and increase the available storage of the database.

[0064] Please refer to Figure 3 , Figure 3is a structural schematic diagram of an embodiment of a database management system of the present application, the database management system 30 comprises: a control module 300 and a data center 302, the control module 300 is used for sensing the change of metadata of a database, and generating a metadata message corresponding to the metadata; wherein the metadata of the database is related to a data table in the database, the metadata message is generated based on structure information of the data table in the database, and the structure information at least includes a database name corresponding to the data table, table information and a change type; the data center 302 is used for receiving the metadata message generated by the control module 300; generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating structure information corresponding to the temporary data table to a storage module 3020; wherein the data center 302 comprises the storage module 3020, the temporary data table is different from a pre-defined data table defined by the data center 302, and the temporary data table in the database can be managed based on the structure information stored in the storage module 3020.

[0065] Further, the above-mentioned database management system 30 can implement the method described in any of the above embodiments, and the related content is described in detail in the above method embodiments, which will not be repeated here.

[0066] In a specific application scenario, please refer to Figure 4 , Figure 4 is a topological structure schematic diagram of an embodiment of a database management system of the present application, the data center 302 further comprises a data service module 3021 and a management service module 3022, the data center 302 interfaces with at least one database, the control module 300 monitors the metadata of the database in real time, when a data table in any database changes, the control module 300 generates a structure information corresponding to the changed data table based on a preset protocol to generate a metadata message, and each database with changed metadata has its own metadata message.

[0067] Further, the control module 300 and the data center 302 transmit the metadata message based on the Kafka message queue, and the metadata message comprises at least one sub-message corresponding to at least one data table in the database, the sub-message is related to a change type, and the change type includes creating a table, editing a table and deleting a table. Wherein, the management service module 3022 is used for publishing an event for the database, the data service module 3021 is used for monitoring the event, and determining the task state of the synchronization task corresponding to the metadata message based on the event. Wherein, the event includes binding a database, editing a database and unbinding a database, and the metadata of the database corresponding to the event changes after the event is implemented.

[0068] Optionally, the data service module 3021 specifically comprises a metadata synchronization module 3023, a listening module 3024, an event generation module 3025 and a saving module 3026, the metadata synchronization module 3023 is used for receiving and parsing the metadata message sent by the control module 300 based on the Kafka message queue, calling the saving module 3026 to make the saving module 3026 store the structure information of the temporary data table into the storage module 3020, the event generation module 3025 is used for generating the event message corresponding to the event after the management service module 3022 publishes the event, and then the listening module 3024 listens to the event message, so as to determine the task state of the synchronization task.

[0069] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an embodiment of an electronic device of the present application, the electronic device 50 comprises a memory 501 and a processor 502 coupled with each other, the memory 501 stores program data (not shown in the figure), and the processor 502 calls the program data to realize the method in any of the above embodiments, and the related content is described in the detailed description of the above method embodiments, which will not be repeated here.

[0070] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of an embodiment of a computer readable storage medium of the present application, the computer readable storage medium 60 stores program data 600, and the program data 600 is executed by a processor to realize the method in any of the above embodiments, and the related content is described in the detailed description of the above method embodiments, which will not be repeated here.

[0071] It should be noted that the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0072] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0073] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0074] The above is only the embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A database management method characterized by, The method comprises: In response to the control module of the database matching perceiving that the metadata of the database changes, the data center receives the metadata message corresponding to the metadata generated by the control module; wherein the metadata of the database is related to the data table in the database, the metadata message is generated based on the structure information of the data table in the database, and the structure information at least includes the database name, table information and change type corresponding to the data table; Generate a synchronization task corresponding to the metadata message, execute the synchronization task to determine the temporary data table in the database, and update the structure information of the temporary data table to the storage module of the data center; wherein the temporary data table is different from the pre-defined data table defined by the data center; Based on the structure information stored in the storage module, the temporary data table in the database is managed in the data center; Wherein, the data center is connected to at least one of the databases, when the data table in any of the databases changes, the control module generates the metadata message based on the preset protocol, the control module and the data center transmit the metadata message based on the Kafka message queue, and the metadata message includes at least one sub-message corresponding to at least one of the data tables in the database, the sub-message is related to the change type, and the change type includes creating table, editing table and deleting table.

2. The database management method of claim 1, wherein The control module monitors the metadata of the database in real time, and each metadata of the database corresponding to the metadata message; The receiving the metadata message corresponding to the metadata generated by the control module comprises: Based on the preset protocol, the control module generates the metadata message.

3. The database management method of claim 2, wherein The receiving the metadata message generated by the control module based on the preset protocol comprises: Call the interface of the control module to obtain the cluster address of the message queue; Based on the cluster address, obtain a plurality of sub-messages in the metadata message, and determine the position index of the sub-message in the message queue.

4. The database management method according to claim 3, characterized by, The generating the synchronization task corresponding to the metadata message, executing the synchronization task to determine the temporary data table in the database, and updating the structure information of the temporary data table to the storage module of the data center comprises: Based on the number of sub-messages in the metadata message, generate a synchronization sub-task corresponding to the metadata message, and send the synchronization sub-task to an idle thread; wherein the number of sub-messages in each synchronization sub-task does not exceed the number threshold; Based on the data table corresponding to the sub-message in the synchronization sub-task and the pre-defined data table, determine the temporary data table in the database, and based on the sub-message corresponding to the temporary data table, update the structure information of the temporary data table in the storage module of the data center.

5. The database management method according to claim 4, characterized by, The temporary data table in the database is determined based on the data table corresponding to the sub-message in the synchronization sub-task and the predefined data table, and the structure information of the temporary data table is updated in the storage module of the data center based on the sub-message corresponding to the temporary data table, comprising: Based on the predefined data table, filter the data table corresponding to the sub-message that is repeated in the predefined data table, and obtain the temporary data table in the database; Traverse all the sub-messages corresponding to the temporary data table, and based on the change type corresponding to the sub-message, call the update mode corresponding to the change type to update the structure information of the temporary data table to the storage module.

6. The database management method according to claim 3, characterized by, The data center also includes a management service module and a data service module; After the synchronization task corresponding to the metadata message is generated, the temporary data table in the database is determined, and the structure information corresponding to the temporary data table is updated to the storage module of the data center, the method further comprises: In response to the management service module publishing an event for the database, the data service module monitors the event and determines the task state of the synchronization task corresponding to the metadata message based on the event; Wherein, the event includes binding database, editing database and unbinding database, and the metadata of the database corresponding to the event changes after the event is implemented.

7. The database management method according to claim 6, characterized by, The task state of the synchronization task corresponding to the metadata message is determined based on the event, comprising: In response to the event being the binding database, start receiving the metadata message corresponding to the newly bound database generated by the control module from the beginning, and start the synchronization task corresponding to the newly bound database; In response to the event being the editing database, pause the synchronization task corresponding to the edited database, and record the position index of the synchronized sub-message in the message queue; In response to the event being the unbinding database, terminate the synchronization task corresponding to the unbound database.

8. The database management method according to any one of claims 1 to 7, characterized by, The temporary data table in the database is managed in the data center based on the structure information stored in the storage module, comprising: Based on the structure information stored in the storage module, the data of the temporary data table and / or the data amount of the temporary data table are viewed in the data center; Based on the structure information stored in the storage module, the deletion instruction for deleting at least part of the temporary data table is sent to the database in the data center.

9. A database management system, characterized by The system comprises: A control module for sensing the change of metadata of the database and generating a metadata message corresponding to the metadata; wherein the metadata of the database is related to the data table in the database, the metadata message is generated based on the structure information of the data table in the database, and the structure information at least includes the database name, table information and change type corresponding to the data table. A data middle platform is configured to receive the metadata message generated by the control module, generate a synchronization task corresponding to the metadata message, execute the synchronization task to determine a temporary data table in the database, and update structure information corresponding to the temporary data table to a storage module. The data middle platform includes the storage module, and the temporary data table is different from a predefined data table defined by the data middle platform. The data middle platform can manage the temporary data table in the database based on the structure information stored in the storage module. The data middle platform interfaces with at least one database. When a data table in any of the databases changes, the control module generates a metadata message based on a preset protocol and structure information corresponding to the changed data table. The control module and the data middle platform transmit the metadata message based on a Kafka message queue. The metadata message includes at least one sub-message corresponding to at least one data table in the database. The sub-message is related to a change type, and the change type includes creating a table, editing a table, and deleting a table.

10. An electronic device, comprising: The program data is executed by the processor to implement the method of any one of claims 1-8. The program data is executed by the processor to implement the method of any one of claims 1-8.

11. A computer readable storage medium having stored thereon program data, wherein, ​

Images