A database operation and maintenance management method and device

Abstract

The present invention discloses a method and device for operation and maintenance management of a database, and relates to the field of computer technology. A specific implementation of the method includes: in response to an operation and maintenance management request for a database instance set, connecting to a database instance in the database instance set to obtain information about stored databases and data tables from the database instance; according to the obtained information about the databases and data tables, respectively aggregating the databases and data tables in the database instance set to obtain a logic library corresponding to the database instance set and a first logic table under the logic library. This implementation can reduce the risk of inaccurate information caused by manual operation, save manpower and time costs, help users automatically maintain logic table and logic library information, solve the problem of data fragmentation, and can associate various relatively independent resources and business information through different dimensions, thereby activating the data core of the entire operation and maintenance management, and has good scalability and flexibility.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a database operation and maintenance management method and device. Background Art

[0002] In the existing solutions, the operation and maintenance logic library is to build an operation and maintenance platform system. Users manually or in the form of loading configuration files enter their own database information (database name, deployment status), table information (some systems do not store table structure) and asset information corresponding to their database instance (host server, cluster, etc.), and then manually edit or upload configuration files to bundle them together as a logic library and logic table to facilitate subsequent operations (such as database DDL operations, monitoring and alarms, etc.). The above existing solutions are all passively informed of key information such as database assets, structures, libraries and tables. Every time the online database-related assets or configuration data changes, the user must know and actively notify the operation and maintenance platform to change the data. When information changes, it may also be due to various reasons (operation and maintenance personnel forget to update data, the uploaded configuration file does not match the actual online configuration, the notification mechanism causes the operation and maintenance platform to not receive the message due to network failure, etc.), which ultimately leads to the inconsistency between the database asset configuration information and the logical library logical table division and the actual online situation, which may cause more serious problems in subsequent operations. For example, because the sharding information under the logical library of the operation and maintenance platform is inconsistent with the actual online configuration, R&D personnel missed several shards or tables when executing DDL operations; operation and maintenance personnel missed key monitoring and alarm information due to incorrect asset configuration information.

[0003] In addition, most current operation and maintenance platforms use traditional DRMS ​​relational databases to store asset configuration and database information, which lacks scalability and flexibility when solving problems such as online database sharding and resharding.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] It requires unnecessary manual operations, carries the risk of inaccurate information, is time-consuming and labor-intensive, and has poor scalability and flexibility. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a database operation and maintenance management method and device, which can reduce the risk of inaccurate information caused by manual operation, save manpower and time costs, help users automatically maintain logical tables and logical library information, solve the problem of data fragmentation, and can associate various relatively independent resources and business information through different dimensions, thereby activating the data core of the entire operation and maintenance management, and has good scalability and flexibility.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for operation and maintenance management of a database is provided.

[0008] A method for operation and maintenance management of a database comprises: in response to an operation and maintenance management request for a database instance set, connecting to a database instance in the database instance set to obtain information of a stored database and a data table from the database instance; and according to the obtained information of the database and the data table, respectively aggregating the database and the data table in the database instance set to obtain a logic library corresponding to the database instance set and a first logic table under the logic library.

[0009] Optionally, according to the acquired information of the database and the data table, the databases and the data tables in the database instance set are aggregated respectively to obtain a logical library corresponding to the database instance set and a first logical table under the logical library, including: for the data tables in the database instance set, aggregating according to the table structure and basic table name of the data table to obtain one or more second logical tables; the basic table name of the data table is determined according to the table name of the data table; determining the association relationship between the second logical table and the database, wherein the database associated with the second logical table is the database where the data table obtained by aggregation is located; according to the basic library name of the database associated with the second logical table, aggregating the databases associated with the same second logical table into the logical library; the basic library name of the database is determined according to the library name of the database; for the data tables corresponding to the same logical library, aggregating according to the table structure and basic table name of the data table to obtain the first logical table under the logical library, wherein the data tables corresponding to the logical library are the data tables in the database obtained by aggregation to obtain the logical library.

[0010] Optionally, the aggregation according to the table structure and basic table name of the data table includes: processing the column information of the data table into a first string, establishing a mapping relationship between the first string and the table name of the data table; aggregating the data tables with the same basic table name and the mapping relationship between the table name and the same first string into one logical table.

[0011] Optionally, the basic table name or the basic library name is determined in the following manner: the second character string is segmented, and a character string without word meaning is extracted through a regular expression using specified special characters in the characters obtained by the segmentation, and the remaining characters after the extraction are concatenated to obtain a third character string; wherein the second character string is the table name of the data table, and the third character string is the basic table name of the data table; or, the second character string is the library name of the database, and the third character string is the basic library name of the database.

[0012] Optionally, after obtaining the logical library corresponding to the database instance set and the first logical table under the logical library, the method includes: constructing a logical structure layer corresponding to the database instance set according to the architecture of the logical library and the architecture of the database, as well as the aggregation relationship between the database and the logical library, and the aggregation relationship between the data table and the first logical table, and storing the logical structure layer in a graph database, wherein the architecture of the logical library includes the relationship between the logical library and the respective first logical tables, and the architecture of the database includes the relationship between the database and the respective data tables.

[0013] Optionally, after obtaining the logical library corresponding to the database instance set and the first logical table under the logical library, it also includes: based on basic resource information of the database instance set, building an infrastructure layer corresponding to the database instance set, and storing the infrastructure layer in the graph database, wherein the basic resource information includes at least information about the database cluster in the database instance set and information about the database instance under the database cluster.

[0014] Optionally, it also includes: mapping the logical structure layer to the infrastructure layer, obtaining a database architecture knowledge graph corresponding to the database instance set, and storing the database architecture knowledge graph in the graph database.

[0015] According to another aspect of an embodiment of the present invention, a database operation and maintenance management device is provided.

[0016] A database operation and maintenance management device comprises: an information acquisition module, which is used to respond to an operation and maintenance management request for a database instance set, connect to a database instance in the database instance set, and obtain information about stored databases and data tables from the database instance; an operation and maintenance management module, which is used to aggregate the databases and data tables in the database instance set respectively according to the obtained information about the databases and data tables, and obtain a logic library corresponding to the database instance set and a first logic table under the logic library.

[0017] Optionally, the operation and maintenance management module is also used to: for the data tables in the database instance set, aggregate according to the table structure and basic table name of the data table to obtain one or more second logical tables; the basic table name of the data table is determined according to the table name of the data table; determine the association relationship between the second logical table and the database, wherein the database associated with the second logical table is the database where the data table obtained by aggregation is located; according to the basic library name of the database associated with the second logical table, aggregate the databases associated with the same second logical table into the logical library; the basic library name of the database is determined according to the library name of the database; for the data tables corresponding to the same logical library, aggregate according to the table structure and basic table name of the data table to obtain the first logical table under the logical library, wherein the data tables corresponding to the logical library are the data tables in the database obtained by aggregation.

[0018] Optionally, the operation and maintenance management module includes an aggregation submodule, which is used to: process the column information of the data table into a first string, establish a mapping relationship between the first string and the table name of the data table; and aggregate the data tables with the same basic table name and the mapping relationship between the table name and the same first string into one logical table.

[0019] Optionally, the operation and maintenance management module includes a basic name determination submodule, which is used to determine the basic table name or the basic library name in the following manner: segmenting the second character string, using specified special characters in the characters obtained by the segmentation, extracting a character string without word meaning through a regular expression, and concatenating the remaining characters after the extraction to obtain a third character string, wherein the second character string is the table name of the data table and the third character string is the basic table name of the data table; or, the second character string is the library name of the database and the third character string is the basic library name of the database.

[0020] Optionally, the operation and maintenance management module is also used to: construct a logical structure layer corresponding to the database instance set according to the architecture of the logical library and the architecture of the database, as well as the aggregation relationship between the database and the logical library, and the aggregation relationship between the data table and the first logical table, and store the logical structure layer in a graph database, wherein the architecture of the logical library includes the relationship between the logical library and the respective first logical tables, and the architecture of the database includes the relationship between the database and the respective data tables.

[0021] Optionally, the operation and maintenance management module is also used to: build an infrastructure layer corresponding to the database instance set based on the basic resource information of the database instance set, and store the infrastructure layer in the graph database, wherein the basic resource information includes at least the information of the database cluster in the database instance set and the information of the database instance under the database cluster.

[0022] Optionally, the operation and maintenance management module is also used to: map the logical structure layer to the infrastructure layer, obtain a database architecture knowledge graph corresponding to the database instance set, and store the database architecture knowledge graph in the graph database.

[0023] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0024] An electronic device comprises: one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the database operation and maintenance management method provided in an embodiment of the present invention.

[0025] According to yet another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0026] A computer-readable medium stores a computer program, which, when executed by a processor, implements the database operation and maintenance management method provided by an embodiment of the present invention.

[0027] One embodiment of the above invention has the following advantages or beneficial effects: in response to an operation and maintenance management request for a database instance set, a database instance in the database instance set is connected to obtain information about the stored database and data table from the database instance; based on the obtained information about the database and data table, the database and data table in the database instance set are aggregated to obtain a logical library corresponding to the database instance set and the first logical table under the logical library. It can reduce the risk of inaccurate information caused by manual operation, save manpower and time costs, help users automatically maintain logical tables and logical library information, solve the problem of data fragmentation, and can associate various relatively independent resources and business information through different dimensions, thereby activating the data core of the entire operation and maintenance management, and has good scalability and flexibility.

[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0030] Figure 1 is a schematic diagram of the main steps of a database operation and maintenance management method according to an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of the operation and maintenance management process of a database according to an embodiment of the present invention;

[0032] Figure 3 It is a schematic diagram of the storage structure of the database architecture knowledge graph according to one embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of obtaining library table information according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of a second logic table obtained by aggregation according to an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of a logic library and a first logic table under the logic library obtained by aggregation according to an embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of main modules of a database operation and maintenance management device according to an embodiment of the present invention;

[0037] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0038] Fig. 9 It is a schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0040] In the era of big data, with the development of business, the amount of data and the number of visits are constantly increasing. The technology of sharding databases and tables has begun to be widely used in databases to break through the I / O (input / output) capacity limitations of single-node database servers and solve database scalability problems. While it brings optimization to system performance, it also adds considerable difficulty to the operation and maintenance of databases. One master and multiple slaves, shared instances, vertical or horizontal sharding databases and tables will bring about a variety of database cluster structures. R&D and operation and maintenance personnel often need to invest a lot of energy to manually maintain database structures, database cluster deployment and other key information. In the era of cloud computing, physical libraries (sharding libraries) and physical tables (sharding tables) are more hidden under the implementation. Users are more concerned about how to operate the logical libraries and logical tables that operate as a database cluster as a whole and how the performance is.

[0041] Figure 1 It is a schematic diagram of main steps of a database operation and maintenance management method according to an embodiment of the present invention.

[0042] like Figure 1 As shown, a database operation and maintenance management method according to an embodiment of the present invention mainly includes the following steps S101 to S102.

[0043] Step S101: in response to an operation and maintenance management request for a database instance set, connecting to a database instance in the database instance set to obtain information of a stored database and a data table from the database instance;

[0044] Step S102: according to the acquired information of the database and the data table, the database and the data table in the database instance set are aggregated respectively to obtain a logic library corresponding to the database instance set and a first logic table under the logic library.

[0045] Operational management requests for a collection of database instances can be triggered based on demand.

[0046] The information of the database and data table stored in the database instance is obtained, which specifically includes all the database, table and column information stored in the database instance.

[0047] In one embodiment, according to the information of the acquired database and data table, the database and data table in the database instance set are aggregated respectively to obtain the logical library corresponding to the database instance set and the first logical table under the logical library, including: for the data tables in the database instance set, the data tables are aggregated according to the table structure and basic table name to obtain one or more second logical tables; the basic table name of the data table is determined according to the table name of the data table; the association relationship between the second logical table and the database is determined, wherein the database associated with the second logical table is the database where the data table where the second logical table is aggregated is located; according to the basic library name of the database associated with the second logical table, the databases associated with the same second logical table are aggregated into a logical library to obtain at least one logical library; the basic library name of the database is determined according to the library name of the database; for the data tables corresponding to the same logical library, the data tables are aggregated according to the table structure and basic table name to obtain the first logical table under the logical library, wherein the data tables corresponding to the logical library are the data tables in the database where the logical library is aggregated. The above-mentioned database used to aggregate to obtain the logical library can also be called a physical library, and the data tables used to aggregate to obtain the logical table can also be called a physical table.

[0048] In one embodiment, the data table is aggregated according to the table structure and the basic table name, including: processing the column information of the data table into a string of column information, the string is called a first string, and establishing a mapping relationship between the first string and the table name of the data table; aggregating the data tables with the same basic table name and the table name having a mapping relationship with the same first string into a logical table. Aggregation to obtain the first logical table or the second logical table can be aggregated according to the aggregation process.

[0049] In one embodiment, the basic table name or basic library name is determined in the following manner: the second character string is segmented, and a character string without word meaning is extracted through a regular expression using specified special characters in the characters obtained by the segmentation, and the remaining characters after the extraction are concatenated to obtain a third character string; wherein the second character string is the table name of the data table, and the third character string is the basic table name of the data table; or, the second character string is the library name of the database, and the third character string is the basic library name of the database.

[0050] In one embodiment, after obtaining the logic library corresponding to the database instance set and the first logic table under the logic library, it includes: building a logical structure layer corresponding to the database instance set according to the architecture of the logic library and the architecture of the database, as well as the aggregation relationship between the database and the logic library, and the aggregation relationship between the data table and the first logic table, and storing the logical structure layer in a graph database, wherein the architecture of the logic library includes the relationship between the logic library and the respective first logic tables, and the architecture of the database includes the relationship between the database and the respective data tables.

[0051] In one embodiment, after obtaining the logical library corresponding to the database instance set and the first logical table under the logical library, it also includes: based on the basic resource information of the database instance set, building an infrastructure layer corresponding to the database instance set, and storing the infrastructure layer in the graph database, wherein the basic resource information includes at least the information of the database cluster in the database instance set and the information of the database instance under the database cluster.

[0052] In one embodiment, the logical structure layer can also be mapped to the infrastructure layer to obtain a database architecture knowledge graph corresponding to the database instance set, and the database architecture knowledge graph is stored in the graph database. The database architecture knowledge graph includes the above-mentioned logical structure layer and infrastructure layer.

[0053] The operation and maintenance management method of the database of the embodiment of the present invention obtains the database information and table structure stored in the database instance in each cluster by active exploration, and divides the tables and libraries of the same structure into logical tables and logical libraries through a series of rule definitions and algorithm analysis, thereby realizing automatic exploration of sub-library and sub-table structures, logical tables and logical libraries, and timely correcting inaccurate data. The data is stored in the form of a graph in a graph database, which can flexibly cope with various sub-library and sub-table structures and subsequent expansions. This can help R&D and operation and maintenance personnel save unnecessary manual operations, and avoid the problem of inaccurate information caused by human operations. It greatly saves manpower and time costs, and reduces online operation and maintenance risks as much as possible.

[0054] Figure 2 This is a schematic diagram of the operation and maintenance management process of a database according to an embodiment of the present invention. The embodiment of the present invention can automatically integrate a multi-cluster database with separate libraries and tables into a logical library by actively exploring. The process includes traversing the project. A project can include one or more database clusters (referred to as clusters). A database cluster includes database instances (referred to as instances). The set of database instances under a project is a database instance set. For the traversed project, perform the following steps:

[0055] The first step is to obtain all cluster instances online;

[0056] The second step is to connect to an instance in the cluster (such as a MySQL instance) and obtain all library and table information stored in the instance through the information_schema library in the MySQL instance. The library and table information specifically includes the library, the tables contained in the library, and the column information contained in the table.

[0057] The third step is to integrate the processed library, table, and column information, including: aggregating tables with the same table structure and similar or identical table names together to form a logical table named by the basic table name; aggregating databases that contain sub-tables of the logical table together to form a logical library named by the basic library name;

[0058] The fourth step is to integrate the data obtained in the above three steps and the relationships between the data, and store them in the graph database to form a knowledge graph.

[0059] In order to store the complex and ever-changing architecture in the sub-libraries and sub-tables, preserve the complex relationships between entities and the future scalability based on business needs, an embodiment of the present invention uses a graph to store the relationships between online projects, databases and assets, that is, the knowledge graph obtained in the fourth step above, or the database architecture knowledge graph.

[0060] The storage structure of the database architecture knowledge graph of an embodiment of the present invention can be specifically as follows: Figure 3 As shown, according to Figure 3 In the example, a project includes two database clusters: database cluster 1_1 (DBCluster1_1) and database cluster 1_2 (DBCluster1_2). Each database cluster includes three database instances (instance1_1, instance1_2, instance1_3). The database clusters and their instances belong to the asset layer (or infrastructure layer). Database cluster 1_1 stores Scatter Database_1, and database cluster 1_2 stores Scatter Database_2. Both Scatter Database_1 and Scatter Database_2 are physical databases. Scatter Database_1 includes Scatter Table 1_1 and Scatter Table 2_1. Scatter Database_2 includes Scatter Table 1_2 and Scatter Table 2_2. These Scatter Tables are physical tables. Scatter Table 1_1 and Scatter Table 1_2 are aggregated into Logical Table 1, and Scatter Table 2_1 and Scatter Table 2_2 are aggregated into Logical Table 2. Logical Table 1 and Logical Table 2 both belong to Logical Database. The relationship between a logical database and its logical tables, the relationship between a database and its data tables, the aggregation relationship between a database and a logical database, and the aggregation relationship between a data table and a logical table belong to the business layer (or logical structure layer).

[0061] The following is a detailed description of the database operation and maintenance management method according to an embodiment of the present invention.

[0062] First, a scope is defined to determine which database instances are included in the database instance set. In one embodiment of the present invention, the database instance set is a set of all database instances belonging to the same project. In other embodiments, the scope of the database instance set can also be adjusted according to the business. For example, the database instance set can be a set of database instances belonging to the same department, or the same business party or individual. The present invention does not limit this.

[0063] Taking the database instance set as the set of all database instances belonging to the same project as an example, the embodiment of the present invention performs automatic aggregation calculation of logical libraries and logical tables on all database instances belonging to the same project, while for database instances that do not belong to the same project, no logical library aggregation calculation is performed.

[0064] In the MySQL database, there is a built-in information_schema library, which stores all other database information maintained by the MySQL server, such as the database name, database tables, data types and access permissions of table columns, etc. In order to obtain the database and its table structure, the following tables (SCHEMATA, TABLES, COLUMNS) will be used:

[0065] SCHEMATA: Used to provide information about all databases in the current MySQL instance. The results of show databases come from this table.

[0066] TABLES: Provides information about tables in the database (including views). It details the schema (library) to which a table belongs, the table type, table engine, creation time, etc. It is the source of the show tables result.

[0067] COLUMNS: Used to provide column information in a table, which details all the columns of a table and the information of each column.

[0068] By accessing these three tables in sequence, all libraries stored in a MySQL instance, the tables contained in the library, and all the column information of the table can be retrieved and saved in the object in sequence. All libraries stored in a MySQL instance, the tables contained in the library, and all the column information of the table are specific examples of the information of the database and data table stored in the database instance.

[0069] After the above data is retrieved, it needs to be processed. For the column information corresponding to the table, after removing the invalid information, the remaining information is spliced ​​into a string of column information. Invalid information, for example, information related to comment attributes (comment) and default attributes (default) carried in the column information, such as "COMMENT'Update time'", "DEFAULT'vitess_gate_pod_a'" and other similar information are invalid information and need to be removed when processing the column information.

[0070] The string of column information obtained by concatenation is stored as a key in the Map, and the value corresponding to the key is a list of table objects (which stores the table name and the database name to which the table belongs). In this way, all table objects with the same table structure can be stored in a list. As shown below:

[0071] Map[table_struct]=[table_object_1,table_object_2...]

[0072] According to the above expression, the column information of the data table represents the table structure (table_struct). Each table structure corresponds to a table. By using the table structure as the key and the table name as the value, you can find the table name through the table structure.

[0073] According to an embodiment of the present invention, for a data table, the column information of the data table is processed (for example, invalid information such as comments are removed and then spliced), and a character string of the column information of the data table (i.e., the first character string) can be obtained. Therefore, table_struct can be the processed column information, i.e., the character string of the column information, so that the above expression establishes a mapping relationship between the character string of the column information (i.e., the first character string) and the table name of the data table.

[0074] For a logical table, the structural information of all its sub-tables is sufficient but not a necessary condition. In actual business, especially for tables with simple table structures, there are situations where the table structures are the same but the data objects represented are completely different; or some tables store current data, and some tables store historical data. Therefore, in view of the characteristics of sub-libraries and sub-tables, the embodiment of the present invention also adds a similarity judgment on the table names. In sub-libraries and sub-tables, the sub-table names are mostly named by adding numbers or letters to the basic table name, for example (where "x" represents a character):

[0075] [xxx_xxx_1,xxx_xxx_2…],[xxx_1_xxx,xxx_2_xxx…],[xxx_xxx_a,xxx_xxx_b…]

[0076] Therefore, tokenization is required when processing these table names. Use the underscore, hyphen and other special characters in the segmented characters as the demarcation, use regular expressions to extract the strings that do not have any meaning (such as numbers or some single characters), and then reassemble the remaining strings into a basic table name. For example, the table names are: good_order_1, good_order_2, then the special characters are "_", "1", and "2". Use these three characters to construct a regular expression, find the strings that do not have any meaning, namely "_1" and "_2", extract them, and then reassemble the remaining strings into a third string, namely: good_order, to get the basic table name.

[0077] Only tables with the same table structure and the same basic table name will be considered to belong to the same logical table. The basic table name is the table name of this logical table. However, this logical table is also temporary. The temporary logical table can be called the second logical table. The process of aggregating to obtain the second logical table specifically includes: taking the database instance as a MySQL instance as an example, for the data table stored in the MySQL instance in the database instance set, the column information of the data table is processed into a string of column information (first string), and a mapping relationship between the first string and the table name of the data table is established according to the method described above. Since the column information represents the table structure, according to the mapping relationship, the table structure can be used to find the table name with the same table structure. Then, the data tables with the same basic table name and the table name that has a mapping relationship with the same first string are aggregated into one logical table, that is, the second logical table is obtained.

[0078] When the aggregate table object becomes a logical table, the database corresponding to each sub-table object will be stored in a list, mapped by the basic table name of these sub-tables as the key. Similar to the previous aggregate sub-table, all databases that may contain this logical table are also aggregated in a list, for example, in the following format:

[0079] Map[table_base_name]=[db1,db2,db3……]

[0080] Similar to the logic of the aggregated logical table, databases with the same logical table can only be regarded as sufficient but not necessary conditions for forming a logical library. Some databases may have exactly the same table information, but the functions and stored data are completely different, such as the historical database (a database that stores historical data). Therefore, it is necessary to extract the basic library name through library name processing. The method of extracting the basic library name is the same as the method of extracting the basic table name, that is, when processing these library names, tokenize the library names, use the underscore, hyphen and other special characters in the split characters as the division, use regular expressions to extract the strings without any meaning (such as numbers or some single characters), and then reassemble the remaining strings into a basic library name.

[0081] Only databases with the same logical table and the same basic library name can be aggregated into a logical library. Once the logical library is calculated, the range of sub-tables contained in the logical table can be reversed to obtain the final logical table. That is, the above temporary logical table (i.e., the second logical table) needs to be divided according to the logical library before the final logical table can be truly determined. The final logical table can be called the first logical table.

[0082] The first logical table can be obtained by aggregation as follows: since the second logical table is aggregated from some data tables, there is an association relationship between the databases where these data tables are located and the second logical table. According to the basic library name of the database associated with the second logical table, the databases associated with the same second logical table are aggregated into a logical library. Specifically, the databases with the same basic library name and associated with the same second logical table are aggregated into the same logical library. The data tables contained in the databases obtained by aggregation and the same logical library have a corresponding relationship with the logical library, that is, the data table is the data table corresponding to the logical library. For the data table corresponding to the same logical library, the column information of the data table is processed into a string of the column information of the data table (for ease of description, the string of the column information is called the first string), and the mapping relationship between the first string and the table name of the data table is established with reference to the method described above. Since the column information represents the table structure, according to the mapping relationship, the table name with the same table structure can be found using the table structure. Then, the data tables with the same basic table name and the table name with the same first string are aggregated into a logical table, that is, the first logical table under the logical library is obtained.

[0083] The program that completes the above logic is called KBMS (KBMS), which is further introduced below with a specific example.

[0084] First, the KBMS of the embodiment of the present invention receives a computing logic library request (i.e., an operation and maintenance management request for a set of database instances, referred to as a request), which is accompanied by a project number. In response to the request, the KBMS links to the slave library instances of all database clusters under this project (this can reduce the impact on the online), and obtains the information of all databases and data tables of the slave library instances (referred to as library and table information) through the information_schema library. The schematic diagram of obtaining library and table information in one embodiment of the present invention is shown in FIG. Figure 4 As shown, there are three order libraries (order databases), namely orders-1, orders-2 and orders-history; these three libraries each have an order table (order data table), namely good_order_1, good_order_2 and good_order.

[0085] Since the data structures of the three order tables are the same and their basic table names are the same (all good_order), the three tables will be aggregated into a logical table good_order (i.e., the second logical table), and the databases associated with the original three tables will be associated with this logical table, that is, the databases associated with this logical table are orders-1, orders-2, and orders-history. The schematic diagram of the aggregated second logical table is as follows: Figure 5 shown.

[0086] Figure 5 In the example, although orders-1, orders-2, and orders-history are all associated with good_order, the base library names of orders-1 and orders-2 (all orders) are different from the base library name of orders-history (orders-history). Therefore, according to their base library names, they will be divided into two logical libraries when calculating the logical library: orders and orders-history. After obtaining the final logical library, the actual logical table (i.e., the first logical table) is divided by associating the sub-libraries and sub-tables downwards. The schematic diagram of the logical library and the first logical table under the logical library is shown in the figure below. Figure 6 As shown, there is a first logical table good_order under the logical library orders and orders-history respectively, but these two first logical tables good_order are aggregated from different data tables. The first logical table good_order under the logical library orders is aggregated from the data tables good_order_1 and good_order_2, and the first logical table good_order under the logical library orders-history is aggregated from the data table good_order.

[0087] In addition, the information about which database clusters (clusters for short) the logical libraries are related to and on which database instances (instances for short) the sub-libraries under the logical libraries are deployed is recorded when KBMS first accesses the instance of each cluster. These node information, master-slave architecture, etc. will be packaged together by KBMS and sent to the data storage unit for storage. The data storage unit is a unit used to store this data.

[0088] After calculating the logic library and logic table, an appropriate data structure is also needed to store its architecture information. Due to the flexible and changeable database structure and deployment relationship of the logic library, and the association across multiple entities, the embodiment of the present invention adopts the form of graph for storage. Its model structure is as follows: Figure 3 As shown, this is the storage structure of the database architecture knowledge graph.

[0089] Based on online operation and maintenance experience and models, the logical library can be divided into two categories: business layer and asset layer for graph construction:

[0090] Asset layer (i.e. infrastructure layer): includes basic resource information such as database clusters, database instances, containers, and physical machines. These are the foundation for building upper-layer businesses. This layer mainly records the configuration information of hardware resources and basic software resources (such as operating systems and database systems). It reflects the deployment architecture information.

[0091] Business layer (i.e. logical structure layer): includes the architecture information of the logical library, the library table structure information, etc. It reflects how a database business is designed and how it is logically constructed.

[0092] Specifically, the logical structure layer can be constructed according to the architecture of the logic library and the architecture of the database, as well as the aggregation relationship between the database and the logic library, and the aggregation relationship between the data table and the first logic table. The architecture of the logic library includes the relationship between the logic library and the respective first logic tables, and the architecture of the database includes the relationship between the database and the respective data tables.

[0093] Based on the basic resource information of the database instance set, an infrastructure layer corresponding to the database instance set is constructed.

[0094] After obtaining the logical library information transmitted from KBMS, the business layer and the asset layer can be independently constructed. Finally, through the deployment information, the relationship between the business layer and the asset layer is asynchronously constructed, and the business logic is mapped to the infrastructure (that is, the logical structure layer is mapped to the infrastructure layer). The database architecture knowledge graph is obtained and stored in the graph database. The advantage of this is that the complex and flexible relationship can be split into two simple and independent graphs for construction, which reduces the problem of uncontrollable and time-consuming construction process due to too many relationship loops in graph construction. In addition, using graph storage can solve the problem of data fragmentation in operation and maintenance, and associate various relatively independent resources and business information through different dimensions, thereby activating the data core of the entire operation and maintenance management.

[0095] The embodiment of the present invention realizes active exploration of the database structure and automatic identification of the logical library and logical table, and obtains and constructs the database architecture information and database storage information by actively linking the database, thereby reducing the risks brought by manual operation and saving manpower and time costs. Based on the principle of the method of sub-library and sub-table, by comparing the data structure information between different library tables, a group of libraries (physical libraries) and tables (physical tables) with extremely high similarity are found, and the corresponding identification is as logical library and logical table, which helps users to automatically maintain the logical table and logical library information. The method of storing complex database architecture in the form of knowledge graph and simplifying its construction process solves the problem of data fragmentation, and associates various relatively independent resources and business information through different dimensions, thereby activating the data core of the entire operation and maintenance management. The embodiment of the present invention also saves time and effort for large clusters with more than hundreds of sub-libraries or sub-tables.

[0096] Figure 7 It is a schematic diagram of main modules of a database operation and maintenance management device according to an embodiment of the present invention.

[0097] like Figure 7 As shown, a database operation and maintenance management device 700 according to an embodiment of the present invention mainly includes: an information acquisition module 701 and an operation and maintenance management module 702 .

[0098] The information acquisition module 701 is used to connect to the database instance in the database instance set in response to the operation and maintenance management request for the database instance set to obtain the information of the stored database and data table from the database instance;

[0099] The operation and maintenance management module 702 is used to aggregate the databases and data tables in the database instance set according to the acquired information of the databases and data tables, and obtain a logic library corresponding to the database instance set and a first logic table under the logic library.

[0100] The operation and maintenance management module 702 can be specifically used for: for data tables in a database instance set, aggregating them according to the table structure and basic table name of the data table to obtain one or more second logical tables; determining the basic table name of the data table according to the table name of the data table; determining the association relationship between the second logical table and the database, wherein the database associated with the second logical table is the database where the data table obtained by aggregation is located; aggregating the databases associated with the same second logical table into a logical library according to the basic library name of the database associated with the second logical table; determining the basic library name of the database according to the library name of the database; for the data tables corresponding to the same logical library, aggregating them according to the table structure and basic table name of the data table to obtain the first logical table under the logical library, wherein the data table corresponding to the logical library is the data table in the database obtained by aggregation.

[0101] The operation and maintenance management module 702 may include an aggregation submodule, which is used to: process the column information of the data table into a first string, establish a mapping relationship between the first string and the table name of the data table; aggregate data tables with the same basic table name and a mapping relationship between the table name and the same first string into a logical table.

[0102] The operation and maintenance management module 702 may include a basic name determination submodule, which is used to determine the basic table name or the basic library name in the following manner: segment the second character string, use specified special characters in the characters obtained by the segmentation, extract the character string without word meaning through a regular expression, and concatenate the remaining characters after the extraction to obtain a third character string, wherein the second character string is the table name of the data table and the third character string is the basic table name of the data table, or the second character string is the library name of the database and the third character string is the basic library name of the database.

[0103] The operation and maintenance management module 702 can also be used to: build a logical structure layer corresponding to the database instance set according to the architecture of the logical library and the architecture of the database, as well as the aggregation relationship between the database and the logical library, and the aggregation relationship between the data table and the first logical table, and store the logical structure layer in the graph database, wherein the architecture of the logical library includes the relationship between the logical library and the respective first logical tables, and the architecture of the database includes the relationship between the database and the respective data tables.

[0104] The operation and maintenance management module 702 can also be used to: build an infrastructure layer corresponding to the database instance set based on the basic resource information of the database instance set, and store the infrastructure layer in the graph database, wherein the basic resource information includes at least the information of the database cluster in the database instance set and the information of the database instance under the database cluster.

[0105] The operation and maintenance management module 702 can also be used to: map the logical structure layer to the infrastructure layer, obtain a database architecture knowledge graph corresponding to the database instance set, and store the database architecture knowledge graph in a graph database.

[0106] In addition, the specific implementation content of the database operation and maintenance management device in the embodiment of the present invention has been described in detail in the above-mentioned database operation and maintenance management method, so the repeated content will not be described again here.

[0107] Figure 8 An exemplary system architecture 800 is shown to which the database operation and maintenance management method or database operation and maintenance management device according to the embodiment of the present invention can be applied.

[0108] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0109] Users can use terminal devices 801, 802, and 803 to interact with server 805 through network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0110] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0111] The server 805 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 801, 802, and 803. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0112] It should be noted that the database operation and maintenance management method provided in the embodiment of the present invention is generally executed by the server 805 , and accordingly, the database operation and maintenance management device is generally arranged in the server 805 .

[0113] It should be understood that Figure 8The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0114] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 suitable for implementing a server of an embodiment of the present application. Fig. 9 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0116] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0117] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present application are executed.

[0118] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes an information acquisition module and an operation and maintenance management module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the information acquisition module may also be described as "a module for responding to an operation and maintenance management request for a set of database instances, connecting to a database instance in a set of database instances, and obtaining information about stored databases and data tables from the database instances".

[0121] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: in response to an operation and maintenance management request for a database instance set, connecting to a database instance in the database instance set to obtain information about stored databases and data tables from the database instance; according to the obtained information about the databases and data tables, respectively aggregating the databases and data tables in the database instance set to obtain a logic library corresponding to the database instance set and a first logic table under the logic library.

[0122] According to the technical solution of the embodiment of the present invention, in response to the operation and maintenance management request for the database instance set, the database instance in the database instance set is connected to obtain the information of the stored database and data table from the database instance; according to the obtained information of the database and data table, the database and data table in the database instance set are respectively aggregated to obtain the logical library corresponding to the database instance set and the first logical table under the logical library. It can reduce the risk of inaccurate information caused by manual operation, save manpower and time costs, help users automatically maintain the information of logical tables and logical libraries, solve the problem of data fragmentation, and can associate various relatively independent resources and business information through different dimensions, so as to activate the data core of the entire operation and maintenance management, and has good scalability and flexibility.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A database operation and maintenance management method, characterized in that: include: In response to an operation and maintenance management request for a database instance set, connecting to a database instance in the database instance set to obtain information of a stored database and a data table from the database instance; According to the acquired information of the database and the data table, the databases and the data tables in the database instance set are aggregated respectively to obtain a logic library corresponding to the database instance set and a first logic table under the logic library; wherein the tables and libraries with the same structure are grouped together to be defined as a logic table and a logic library; The step of respectively aggregating the databases and data tables in the database instance set according to the acquired information of the databases and data tables to obtain a logic library corresponding to the database instance set and a first logic table under the logic library includes: For the data tables in the database instance set, aggregating according to the table structure and basic table name of the data table to obtain one or more second logical tables; the basic table name of the data table is determined according to the table name of the data table; Determine an association relationship between the second logical table and a database, wherein the database associated with the second logical table is a database where the data table aggregated to obtain the second logical table is located; According to the basic library name of the database associated with the second logical table, the databases associated with the same second logical table are aggregated into the logical library; the basic library name of the database is determined according to the library name of the database; For the data tables corresponding to the same logic library, aggregation is performed according to the table structure and basic table name of the data tables to obtain the first logic table under the logic library, wherein the data table corresponding to the logic library is the data table in the database that is aggregated to obtain the logic library.

2. The method according to claim 1, characterized in that The aggregation according to the table structure and basic table name of the data table includes: Processing the column information of the data table into a first character string, and establishing a mapping relationship between the first character string and the table name of the data table; The data tables having the same basic table name and the mapping relationship between the table name and the same first character string are aggregated into one logical table.

3. The method according to claim 1, characterized in that The basic table name or the basic library name is determined in the following manner: The second character string is segmented, and a character string without word meaning is extracted through a regular expression using designated special characters among the characters obtained by the segmentation, and the remaining characters after the extraction are concatenated to obtain a third character string; wherein the second character string is the table name of the data table, and the third character string is the basic table name of the data table; or, the second character string is the library name of the database, and the third character string is the basic library name of the database.

4. The method according to claim 1, characterized in that After obtaining the logic library corresponding to the database instance set and the first logic table under the logic library, the method includes: According to the architecture of the logic library and the architecture of the database, as well as the aggregation relationship between the database and the logic library and the aggregation relationship between the data table and the first logic table, a logical structure layer corresponding to the set of database instances is constructed, and the logical structure layer is stored in a graph database, wherein the architecture of the logic library includes the relationship between the logic library and the respective first logic tables, and the architecture of the database includes the relationship between the database and the respective data tables.

5. The method according to claim 4, characterized in that After obtaining the logic library corresponding to the database instance set and the first logic table under the logic library, the method further includes: Based on the basic resource information of the database instance set, an infrastructure layer corresponding to the database instance set is constructed, and the infrastructure layer is stored in the graph database, wherein the basic resource information includes at least information about the database cluster in the database instance set and information about the database instance under the database cluster.

6. The method according to claim 5, characterized in that Also includes: The logical structure layer is mapped to the infrastructure layer to obtain a database architecture knowledge graph corresponding to the database instance set, and the database architecture knowledge graph is stored in the graph database.

7. A database operation and maintenance management device, characterized in that: include: An information acquisition module, configured to, in response to an operation and maintenance management request for a database instance set, connect to a database instance in the database instance set to acquire information of a stored database and a data table from the database instance; An operation and maintenance management module, configured to aggregate the databases and data tables in the database instance set according to the acquired information of the databases and data tables, and obtain a logic library corresponding to the database instance set and a first logic table under the logic library; wherein tables and libraries of the same structure are grouped together and defined as logic tables and logic libraries; The step of respectively aggregating the databases and data tables in the database instance set according to the acquired information of the databases and data tables to obtain a logic library corresponding to the database instance set and a first logic table under the logic library includes: For the data tables in the database instance set, aggregating according to the table structure and basic table name of the data table to obtain one or more second logical tables; the basic table name of the data table is determined according to the table name of the data table; Determine an association relationship between the second logical table and a database, wherein the database associated with the second logical table is a database where the data table aggregated to obtain the second logical table is located; According to the basic library name of the database associated with the second logical table, the databases associated with the same second logical table are aggregated into the logical library; the basic library name of the database is determined according to the library name of the database; For the data tables corresponding to the same logic library, aggregation is performed according to the table structure and basic table name of the data tables to obtain the first logic table under the logic library, wherein the data table corresponding to the logic library is the data table in the database that is aggregated to obtain the logic library.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images