A dynamic database and table splitting method based on service requirements

Abstract

The application provides a dynamic database and table division method based on business requirements, relates to the technical field of data processing, and comprises the following steps: acquiring business requirement information and data information in an original database, performing feature extraction, and generating a requirement-data feature set; obtaining matched database and table division rules in a rule database based on the requirement-data feature set and in combination with a preset feature-rule matching table; allocating corresponding databases and table structures to original data and to-be-written data in the original database based on the database and table division rules, and establishing a configuration relationship between the data and the databases and table structures; and performing database and table division on the original database based on the database and table division rules and the configuration relationship, and optimizing the data storage structure and data query strategy after the database and table division in combination with the business requirement information. The application adjusts the configuration of a database connection pool in combination with real-time changing business requirement information, thereby improving the storage and query performance of the database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a dynamic database and table splitting method based on business requirements. BACKGROUND

[0002] A database connection pool is a buffer pool used to store pre-created database connections. When an application needs to communicate with a database, it can obtain a connection from the connection pool instead of creating a connection every time. Therefore, a database connection pool is suitable for applications that need to frequently interact with a database, especially in a high-concurrency environment, which can effectively reduce the overhead of connection creation and destruction, and improve the stability and performance of the system.

[0003] With the rapid development of business, the configuration of a single fixed database connection pool has been difficult to meet the needs of business development. With the change in concurrency, the database connection configuration has to be adjusted. However, many systems run uninterruptedly 7*24 hours, and each modified connection pool configuration must restart the service. Each downtime will affect the smooth operation of the business system to some extent, and bring heavy operation and maintenance cost. This mode is relatively cumbersome and difficult to flexibly follow the rhythm of the business system.

[0004] Therefore, the present application provides a dynamic database and table splitting method based on business requirements. SUMMARY

[0005] The present application provides a dynamic database and table splitting method based on business requirements, which adjusts the configuration of the database connection pool by combining real-time changing business requirement information, thereby improving the storage and query performance of the database and ensuring the stable operation of the system.

[0006] The present application provides a dynamic database and table splitting method based on business requirements, comprising:

[0007] Step 1: Obtain business requirement information and data information in the original database, extract features from the business requirement information and the data information, and generate a requirement-data feature set;

[0008] Step 2: Based on the requirement-data feature set, and combining a preset feature-rule matching table, obtain a matched database and table splitting rule in the rule database;

[0009] Step 3: Based on the database and table splitting rule, allocate corresponding databases and table structures to the original data and the to-be-written data in the original database, and at the same time, establish a configuration relationship between the original data and the to-be-written data and the corresponding databases and table structures;

[0010] Step 4: based on the database and table splitting rule and the configuration relationship, the original database is split into databases and tables, and the data storage structure and data query strategy after database and table splitting are optimized in combination with the business requirement information.

[0011] Preferably, in step 1, it comprises:

[0012] Obtaining the category requirement of the business and the performance requirement under the corresponding category, and generating the business requirement information based on the category requirement and the corresponding performance requirement;

[0013] Preferably, in step 1, it further comprises:

[0014] Obtaining the category information, data structure and performance parameter of the original database, and comprehensively generating the data information;

[0015] Preferably, in step 1, it further comprises:

[0016] The category information comprises: a relational database, a non-relational database, a document database and a graph database.

[0017] The data structure comprises: a table structure, an index and a database management system structure.

[0018] Preferably, in step 1, it further comprises:

[0019] Feature extraction is performed on the business requirement information to generate a first feature set, and feature extraction is performed on the data information to generate a second feature set;

[0020] The first feature set and the second feature set are matched to output a requirement-data feature set.

[0021] Preferably, in step 2, it comprises:

[0022] In the requirement-data feature set, the features consistent with the requirement instruction are selected to construct a to-be-matched feature set;

[0023] Based on the to-be-matched feature set, the corresponding rule matching information is selected from the preset feature-rule matching table;

[0024] Based on the rule matching information, the corresponding database and table splitting rule is selected from the rule database.

[0025] Preferably, in step 3, it comprises:

[0026] Obtaining the database and table splitting rule and performing analysis to obtain a database rule, a table rule and a database and table mapping relationship;

[0027] Obtaining original data in the original database and corresponding data types and data structures, and outputting first data information;

[0028] At the same time, obtaining the data type and performance requirement of the to-be-written data corresponding to the business requirement information, and outputting second data information;

[0029] Based on the first data information and the second data information, and in combination with a preset information-index matching table, a corresponding rule matching index is obtained;

[0030] Based on the first data information, the second data information, and the corresponding rule matching index, data-index allocation information is generated;

[0031] Based on the data-index allocation information, a corresponding allocation instruction is selected in an instruction database;

[0032] Based on the data-index allocation information and the allocation instruction, corresponding database rule and table rule are allocated to the first data information and the second data information;

[0033] At the same time, based on the data-index allocation information, and in combination with a system selection instruction and a manual selection instruction, a target database server is selected in a server candidate library;

[0034] Based on the database rule and the table rule, a sub-database and sub-table allocation result under the target database server is outputted;

[0035] Based on the sub-database and sub-table allocation result, a data-sub-table mapping relationship between the first data information, the second data information, and the sub-database and sub-table is established;

[0036] In combination with the database and table mapping relationship and the data-sub-table mapping relationship, a configuration file under the corresponding mapping relationship is obtained through a preset mapping-configuration matching module;

[0037] Based on the configuration file, a configuration relationship between the first data information, the second data information, and the sub-database and sub-table is established.

[0038] Preferably, in step 4, it includes:

[0039] The business requirement information is monitored in real time, and in combination with the time stamp corresponding to the business requirement information, a business requirement acquisition log is generated;

[0040] The clustering analysis is performed on all the business requirement information in a preset time period in the business requirement acquisition log, and a requirement clustering result is outputted;

[0041] The service demand information is content-analyzed to obtain service demand change data belonging to the first type in a preset period, and service first change information is generated;

[0042] Meanwhile, service demand change data belonging to the second type in the preset period is obtained from the service demand information, and service second change information is generated;

[0043] The demand clustering result in the preset period, all service first change information and service second change information are statistically analyzed, and the service demand change information is output in combination with the time stamp corresponding to the preset period;

[0044] The database connection pool is monitored in real time by the preset monitoring module, the latest configuration parameters of the database are obtained, and the first configuration information is output;

[0045] Based on the service demand change information and the database-table separation rule, and in combination with the latest configuration parameters, a configuration modification instruction is selected from the instruction database;

[0046] Based on the configuration modification instruction, the configuration relationship corresponding to the service demand change information is modified, and configuration parameter modification information is generated;

[0047] Based on the configuration parameter modification information, configuration port information is obtained from the configuration-port correspondence table, and the port state corresponding to the configuration port information is switched to a modification state;

[0048] The configuration parameters corresponding to the port in the modification state are modified, the first object corresponding to the original configuration parameters is deleted, and the second object is generated in combination with the first configuration information;

[0049] The connection relationship between the second object and the first configuration information is established, and the configuration relationship is updated;

[0050] Based on the updated configuration relationship, and in combination with the database-table separation rule, the database-table separation strategy is adjusted, and the data storage structure of the original database is updated based on the adjusted database-table separation strategy.

[0051] Preferably, the data storage structure and data query strategy after database-table separation are optimized in combination with the service demand information, including:

[0052] According to the concurrent demand information in the service demand information, the capacity optimization value of the database connection pool is determined;

[0053] ;

[0054] Wherein, represents the capacity optimization value; represents the maximum number of connections set to the database connection pool, represents the maximum number of connections determined based on the concurrency demand information, represents the performance evaluation factor corresponding to the maximum number of connections; represents the reference value corresponding to ; represents the minimum number of connections set to the database connection pool; represents the minimum number of connections determined based on the concurrency demand information; represents the performance evaluation factor corresponding to the minimum number of connections; represents the reference value corresponding to ; , represents the capacity incompatibility coefficient based on the maximum number of connections and the minimum number of connections;

[0055] Obtain the first data storage structure before sharding and the second data storage structure after sharding, and compare the first storage structure and the second storage structure to obtain the structure change data;

[0056] Based on the structure change data and in combination with the business demand information, obtain the running parameter change data of the database connection pool, and calculate the performance value of the database connection pool;

[0057]

[0058] wherein, represents the performance value of the database connection pool; represents the exponential function; represents the ratio of the number of reused connections to the total number of connections in the database connection pool determined based on the running parameter change data; represents the ratio of the average value of the increase in connection duration of all connections in the database connection pool based on the running parameter change data to the original average connection duration; represents the incremental connection ratio, represents the parameter value of the database connection pool under the i-th performance evaluation indicator before sharding and table splitting; represents the parameter value of the database connection pool under the i-th performance evaluation indicator after sharding and table splitting; represents the performance evaluation factor under the i-th performance evaluation indicator; represents the total number of performance evaluation indicators; represents the correction factor corresponding to the number n of selected performance evaluation indicators;

[0059] and, ;

[0060] When the capacity optimization value is less than the first preset value and the performance value is less than the second preset value, the to-be-optimized index is determined, and the database connection pool parameter, the data storage structure and the data query strategy are continuously optimized.

[0061] The application provides a dynamic database and table splitting method based on business requirements, which firstly accurately acquires business requirement information and original data structure and stored data information in a database, accurately masters dynamic changes of the business requirements, and then extracts features to obtain a requirement-data feature set; subsequently, the matched database and table splitting rules are selected in a rule database through the obtained feature set, and corresponding databases and table structures are allocated to the original data in the database and the to-be-written data, meanwhile, a corresponding configuration relationship is established, and the real-time update of the configuration relationship is realized in synchronization according to the dynamic changes of the business requirements; finally, the database is split into databases and tables in combination with the business requirement information, and the data storage structure and the data query strategy are continuously optimized, so that the query and storage performance of the database can be continuously improved. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0063] Figure 1 It is a flowchart of a dynamic database and table splitting method based on business requirements provided by the embodiment of the application. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions in the application will be clearly and completely described below in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the application.

[0065] As shown in the figure, the dynamic database and table splitting method based on business requirements provided by the embodiment of the application comprises: Figure 1

[0066] Step 1: acquire business requirement information and data information in an original database, extract features from the business requirement information and the data information, and generate a requirement-data feature set;

[0067] ​Step 2: Obtain the matched sub-database and sub-table rule in the rule database based on the demand-data feature set and in combination with the preset feature-rule matching table;

[0068] Step 3: Based on the sub-database and sub-table rule, allocate the corresponding database and table structure to the original data in the original database and the data to be written, and at the same time, establish the configuration relationship between the original data and the data to be written and the corresponding database and table structure;

[0069] Step 4: Based on the sub-database and sub-table rule and the configuration relationship, perform sub-database and sub-table on the original database, and optimize the data storage structure and data query strategy after sub-database and sub-table in combination with the business demand information.

[0070] In this embodiment, the business demand information is information containing the type demand of the business and the performance demand data under the corresponding type;

[0071] In this embodiment, the original database is the initial database before sub-database and sub-table;

[0072] In this embodiment, the data information is the original data information stored in the original database;

[0073] In this embodiment, the demand-data feature set is a set of extracted features containing business demand information and data information;

[0074] In this embodiment, the preset feature-rule matching table is a table containing the mapping relationship between the extracted features and the corresponding sub-database and sub-table rule, which is pre-set;

[0075] In this embodiment, the rule database is a database containing a large number of sub-database and sub-table rules;

[0076] In this embodiment, the sub-database and sub-table rule is a rule for sub-database and sub-table of the original database according to the business demand information, which is usually pre-set;

[0077] In this embodiment, the original data is the data stored in the original database before sub-database and sub-table;

[0078] In this embodiment, the data to be written is new data to be written corresponding to the business demand information;

[0079] In this embodiment, the configuration relationship is the configuration and mapping relationship between the original data and the data to be written and the database and table structure after structure change;

[0080] In this embodiment, the data storage structure is a new storage structure formed after sub-database and sub-table of the original database;

[0081] In this embodiment, the data query strategy is a strategy for querying data and optimizing queries.

[0082] The implementation principle and beneficial effects of the present embodiment are as follows: the present application first accurately acquires business demand information and original data information stored in a data structure in a database, accurately grasps dynamic changes in business demand, and then extracts features to obtain a demand-data feature set; subsequently, the feature set obtained is used to select matched rules for database and table splitting in a rule database, and corresponding databases and table structures are allocated to original data in the database and data to be written, and meanwhile, a corresponding configuration relationship is established, and real-time updating of the configuration relationship is realized in synchronization according to dynamic changes in business demand; finally, the database is split into databases and tables in combination with business demand information, and data storage structures and data query strategies are continuously optimized, so that the query and storage performance of the database can be continuously improved.

[0083] The dynamic database and table splitting method based on business demand provided by the present embodiment comprises the following steps:

[0084] The types of business demand and performance requirements under corresponding types are acquired, and business demand information is generated based on the types of business demand and the corresponding performance requirements.

[0085] The types of business demand include data management requirements, business process requirements, cross-platform and integration requirements, and user experience requirements.

[0086] In this embodiment, the types of business demand are classified according to different dimensions, so as to quickly determine the type of business demand to which the business demand belongs, thereby improving the access performance and storage performance of the database.

[0087] In this embodiment, the performance requirements are performance requirements of each type of business demand.

[0088] In this embodiment, the data management requirements are demand information related to data storage and management, security and privacy, and data backup and recovery.

[0089] In this embodiment, the business process requirements are demand information related to business workflow, task allocation, and business rule configuration.

[0090] In this embodiment, the cross-platform and integration requirements are demand information related to cross-platform compatibility and system integration of the business.

[0091] In this embodiment, the user experience requirements are demand information that can improve user experience, such as responsive design or multi-language support.

[0092] The implementation principle and beneficial effects of the embodiment are as follows: the application can accurately determine the category information and corresponding performance requirement information of the to-be-processed and processed service by acquiring the category requirement and performance requirement under the corresponding category, improve the resolution of the service, accurately master the requirement dynamic information of each service, and provide convenience for subsequent configuration update and database and table splitting.

[0093] The method for dynamically splitting database and table based on service requirement provided by the embodiment of the application further includes the following steps in step 1:

[0094] The category information, data structure and performance parameter of the original database are acquired, and data information is generated by comprehensive generation;

[0095] The category information includes a relational database, a non-relational database, a document database and a graph database.

[0096] The data structure includes a table structure, an index and a database management system structure.

[0097] In the embodiment, the category information is the category to which the original database belongs.

[0098] In the embodiment, the data structure is the structure information of the original database, for example, a table, a column, a row, a primary key and a foreign key.

[0099] In the embodiment, the performance parameter is the performance data of the database, for example, query performance and storage performance.

[0100] In the embodiment, the relational database is a database using a table structure to organize and manage data, for example, a MySQL, Oracle, SQL Server or the like.

[0101] In the embodiment, the non-relational database is a database not using a traditional table structure to organize and manage data, for example, a column storage database, a key-value pair database or the like.

[0102] In the embodiment, the document database is a database storing data in the form of a document.

[0103] In the embodiment, the graph database is a database storing data in the form of a picture.

[0104] In the embodiment, the table structure is a database organized and managed in the form of a table, and each table usually contains multiple rows of data.

[0105] In the embodiment, the index is a data structure used to speed up data retrieval.

[0106] In this embodiment, the database management system structure: a software system for managing a database, responsible for data storage, retrieval, updating and management, for example, database connection pool, etc.

[0107] The implementation principle and beneficial effects of the present embodiment: the present application can accurately identify the category information of the original database, and accurately obtain the data structure of the database and the performance parameters of the database, thereby improving the identification accuracy of the original database, facilitating subsequent division and structure adjustment of the database according to the category of business demand information, and providing accurate data support for subsequent matching of the database and table division rule.

[0108] The embodiment of the present application provides a dynamic database and table division method based on business demand, and step 1 further comprises:

[0109] The feature extraction is performed on the business demand information to generate a first feature set, and the feature extraction is performed on the data information to generate a second feature set;

[0110] The feature matching is performed on the first feature set and the second feature set to output a demand-data feature set.

[0111] In this embodiment, the first feature set: a set composed of features obtained by feature extraction on business demand information;

[0112] In this embodiment, the second feature set: a set composed of features obtained by feature extraction on data information;

[0113] In this embodiment, the feature matching: matching the features in the first feature set and the second feature set, facilitating subsequent rule matching operation.

[0114] The implementation principle and beneficial effects of the present embodiment: the present application extracts the key data features in the business demand information by feature extraction on the business demand information, and establishes a first feature set, meanwhile, extracts the features of the original database from the corresponding data information of the original database, obtains a feature set of the original database, constructs a second feature set, and then matches the features in the first feature set and the second feature set, thereby ensuring data accuracy and facilitating subsequent database and table division matching.

[0115] The embodiment of the present application provides a dynamic database and table division method based on business demand, and step 2 comprises:

[0116] The demand-data feature set selects the features consistent with the demand instruction to construct a to-be-matched feature set;

[0117] Based on the to-be-matched feature set, the corresponding rule matching information is selected from the preset feature-rule matching table;

[0118] Based on the rule matching information, corresponding database and table division rules are selected in the rule database.

[0119] In this embodiment, the demand instruction is an instruction for selecting corresponding target features in the demand-data feature set;

[0120] In this embodiment, the to-be-matched feature set is a set composed of features selected in the demand-data feature set, which is used for subsequent matching to obtain database and table division rules;

[0121] In this embodiment, the rule matching information is matching information containing the mapping relationship between features and rules, which is obtained in the preset feature-rule matching table according to the to-be-matched feature set.

[0122] The implementation principle and beneficial effects of the embodiment are as follows: the present application first selects features consistent with the demand instruction in the demand-data feature set to obtain the to-be-matched feature set, thereby improving the matching degree between each feature in the to-be-matched feature set and the business demand information, and then selects rule matching information in the preset feature-rule matching table through the to-be-matched feature set, and selects corresponding database and table division rules in the rule database based on the rule matching information, thereby improving the matching precision between the database and table division rules and the business demand information and the original database.

[0123] The embodiment of the present application provides a dynamic database and table division method based on business demand, and step 3 comprises the following steps.

[0124] The database and table division rules are obtained and parsed to obtain database rules, table rules and database and table mapping relationships respectively;

[0125] The original data in the original database and the corresponding data types and data structures are obtained to output first data information;

[0126] Meanwhile, the data types and performance requirements of the to-be-written data corresponding to the business demand information are obtained to output second data information;

[0127] Based on the first data information and the second data information, and in combination with the preset information-index matching table, corresponding rule matching indexes are obtained;

[0128] Based on the first data information, the second data information and the corresponding rule matching indexes, data-index allocation information is generated;

[0129] Based on the data-index allocation information, corresponding allocation instructions are selected in the instruction database;

[0130] Based on the data-index allocation information and the allocation instructions, corresponding database rules and table rules are allocated to the first data information and the second data information.

[0131] Meanwhile, based on the data-index allocation information, and in combination with the system selection instruction and the manual selection instruction, the target database server is selected from the server candidate library;

[0132] Based on the sub-database table allocation rule and the sub-table rule, a sub-database table allocation result corresponding to the target database server is outputted;

[0133] Based on the sub-database table allocation result, a data-sub-table mapping relationship between the first data information, the second data information and the sub-database table is established;

[0134] In combination with the sub-database table mapping relationship and the data-sub-table mapping relationship, a configuration file corresponding to the mapping relationship is obtained through a preset mapping-configuration matching module;

[0135] Based on the configuration file, a configuration relationship between the first data information, the second data information and the sub-database table is established.

[0136] In this embodiment, the sub-database rule is a rule for re-dividing the original database into multiple small databases, which is used to disperse data storage and query pressure, and improve database performance and scalability;

[0137] In this embodiment, the sub-table rule is a rule for dividing a single table into multiple sub-tables, which is used to improve data retrieval speed and improve system maintainability;

[0138] In this embodiment, the sub-database table mapping relationship is a mapping relationship between the sub-database and the sub-table;

[0139] In this embodiment, the first data information is information containing original data in the original database and data types and data structures of the roll;

[0140] In this embodiment, the second data information is information containing data types of data to be written and performance requirements;

[0141] In this embodiment, the preset information-index matching table is a matching table containing a mapping relationship between the first data information, the second data information and the rule matching index, which is pre-set, and is used to match the corresponding rule matching index according to the input first data information or second data information;

[0142] In this embodiment, the rule matching index is an index obtained according to the preset information-index matching table, which is used to allocate the sub-database table rule to the corresponding data;

[0143] In this embodiment, the data-index allocation information contains information of the first data information, the second data information and the corresponding rule allocation index, which is used to select the allocation instruction in the subsequent instruction database;

[0144] In this embodiment, the allocation instruction is obtained in the instruction database, and is used to allocate the sub-database and sub-table rule to the first data information and the second data information.

[0145] In this embodiment, the system selection instruction is a server selection instruction matched by the system automatically.

[0146] In this embodiment, the manual selection instruction is a server selection instruction input manually, and the priority is generally higher than that of the system selection instruction.

[0147] In this embodiment, the server candidate library is a database containing various database server information, and is used to select the matched database server according to the system selection instruction and the manual selection instruction.

[0148] In this embodiment, the target database server is a database server selected from the server candidate library.

[0149] In this embodiment, the sub-database and sub-table allocation result is an allocation result obtained after the target data server is divided according to the sub-database and sub-table rule.

[0150] In this embodiment, the data-sub-table mapping relationship is a mapping relationship between the first data information, the second data information and the divided sub-database and sub-table.

[0151] In this embodiment, the preset mapping-configuration matching module is a module used to obtain the corresponding configuration file according to the input sub-database and sub-table mapping relationship and the data-sub-table mapping relationship, and is preset.

[0152] In this embodiment, the configuration file is a configuration file corresponding to each mapping relationship and generated according to the preset mapping-configuration module, and is used to build the configuration relationship between the data and the database subsequently.

[0153] The implementation principle and beneficial effects of the embodiment are as follows: the sub-database and sub-table rule, the sub-database and sub-table mapping relationship, the rule allocation index allocated to the first data information corresponding to the original database and the second data information corresponding to the business requirement information according to the preset information-index matching table are obtained by deeply analyzing the obtained sub-database and sub-table rule, so that the matching degree between the sub-database and sub-table rule, the business requirement and the database is improved, and then the configuration relationship between the data and the sub-database and sub-table is established by the obtained configuration file, so that the compatibility and adaptability between the sub-database and sub-table rule and the data are improved, thereby ensuring the stable operation of the system.

[0154] The method provided in the embodiment of the application comprises the following steps.

[0155] The business demand information is monitored in real time, and a business demand acquisition log is generated in combination with a time stamp corresponding to the business demand information;

[0156] The business demand acquisition log is monitored in real time, and a business demand acquisition log is generated in combination with a time stamp corresponding to the business demand information;

[0157] The business demand information is monitored in real time, and a business demand acquisition log is generated in combination with a time stamp corresponding to the business demand information;

[0158] At the same time, the business demand information in the preset period is obtained, and the business second change information is generated;

[0159] The demand clustering result, all business first change information and business second change information in the preset period are statistically analyzed, and the business demand change information is output in combination with the time stamp corresponding to the preset period;

[0160] The database connection pool is monitored in real time by the preset monitoring module, the latest configuration parameters of the database are obtained, and the first configuration information is output;

[0161] Based on the business demand change information and the database and table splitting rule, and in combination with the latest configuration parameter, the configuration modification instruction is selected in the instruction database;

[0162] Based on the configuration modification instruction, the configuration relationship corresponding to the business demand change information is modified, and the configuration parameter modification information is generated;

[0163] Based on the configuration parameter modification information, the configuration port information is obtained in the configuration-port correspondence table, and the port state corresponding to the configuration port information is switched to the modification state;

[0164] The configuration parameter corresponding to the port in the modification state is modified, the first object corresponding to the original configuration parameter is deleted, and the second object is generated in combination with the first configuration information;

[0165] The connection relationship between the second object and the first configuration information is established and the configuration relationship is updated;

[0166] Based on the updated configuration relationship, and in combination with the database and table splitting rule, the database and table splitting strategy is adjusted, and the data storage structure of the original database is updated based on the adjusted database and table splitting strategy.

[0167] In this embodiment, the time stamp is the time characteristic of each business demand, which is used to record the time information of each event in the system;

[0168] In this embodiment, the business requirement acquisition log: a log recording each business requirement received by the system according to the timestamp of the business requirement;

[0169] In this embodiment, the preset time period: a pre-set time period, used for analyzing the change data of the business requirements in the fixed time period;

[0170] In this embodiment, the requirement clustering result: a result obtained after clustering analysis of the business requirement information, which can be used to count the types of all business requirements, or the frequency of business requirements in each category, facilitating subsequent data processing;

[0171] In this embodiment, the first type: a type same as the original business requirement type or the original performance requirement type;

[0172] In this embodiment, the first change information of the business: information containing the first type of business requirement change data;

[0173] In this embodiment, the second type: a newly added business requirement type and a newly added performance requirement;

[0174] In this embodiment, the second change information of the business: information containing the second type of business requirement change data;

[0175] In this embodiment, the business requirement change information: information used to represent the change of the business requirement in the preset time period, which can record the dynamic change data of the business requirement information;

[0176] In this embodiment, the preset monitoring module: a module used to monitor the configuration parameters in the database, which is pre-set;

[0177] In this embodiment, the database connection pool: a technology used to manage database connections, which can improve the access efficiency and performance of the database;

[0178] In this embodiment, the first configuration information: the latest configuration parameters of the database obtained through the preset monitoring module;

[0179] In this embodiment, the configuration modification instruction: an instruction obtained by selecting the latest configuration parameters in the instruction database, and used to modify the configuration relationship;

[0180] In this embodiment, the configuration parameter modification information: configuration parameter change information obtained after modifying the configuration relationship;

[0181] In this embodiment, the configuration-port correspondence table: a correspondence table containing the mapping relationship between the configuration parameter modification information and the configuration port information, used to obtain the corresponding configuration port information according to the input configuration parameter modification information, so as to modify the corresponding configuration parameters;

[0182] In this embodiment, the configuration port information: a control port for adjusting configuration parameters;

[0183] In this embodiment, the modification state is a state in which the configuration port can receive external configuration modification information and thus can modify the configuration parameters within the port;

[0184] In this embodiment, the first object is a connection pool object corresponding to the original configuration parameters;

[0185] In this embodiment, the second object: that is, the connection pool object corresponding to the latest configuration parameters;

[0186] The implementation principle and beneficial effects of this embodiment: The present invention monitors business demand information in real time and generates a business demand acquisition log in combination with a timestamp, which can not only accurately grasp the dynamic change data of business demand information, but also facilitate subsequent operation and maintenance; at the same time, the present invention can also monitor and obtain the latest configuration parameters of the database in real time, and update the connection pool object and connection pool configuration according to the latest parameters, thereby reducing the number of restarts of the database connection pool configuration, which not only ensures the normal and stable operation of the business system, but also improves the flexibility of the database connection pool and reduces the subsequent operation and maintenance costs of the database.

[0187] The embodiment of the present invention provides a dynamic database and table sharding method based on business needs, which optimizes the data storage structure and data query strategy after database and table sharding in combination with business demand information, including:

[0188] Determine the optimal capacity value of the database connection pool based on the concurrent demand information in the business demand information;

[0189] ;

[0190] in, represents the capacity optimization value; Indicates the maximum number of connections set to the database connection pool. Indicates the maximum number of connections determined based on concurrent demand information. Indicates the performance evaluation factor corresponding to the maximum number of connections; Represents The corresponding benchmark value; Indicates the minimum number of connections set to the database connection pool; Indicates the minimum number of connections determined based on concurrent demand information; Indicates the performance evaluation factor corresponding to the minimum number of connections; Represents The corresponding benchmark value; 、 Indicates the capacity incompatibility coefficient based on the maximum and minimum number of connections;

[0191] obtain a first data storage structure before the database sharding and table splitting and a second data storage structure after the database sharding and table splitting, and compare the first data storage structure and the second data storage structure to obtain structural change data;

[0192] based on the structural change data and in combination with business demand information, obtain running parameter change data of a database connection pool, and calculate a performance value of the database connection pool;

[0193]

[0194] wherein, represents the performance value of the database connection pool; represents an exponential function; represents a ratio of a number of reused connections to a total number of connections in the database connection pool determined based on the running parameter change data; represents a ratio of an average value of an increase in connection duration of all connections in the database connection pool determined based on the running parameter change data to an original average connection duration; represents an incremental connection ratio, represents a parameter value of the database connection pool under an i-th performance evaluation index before the database sharding and table splitting; represents a parameter value of the database connection pool under the i-th performance evaluation index after the database sharding and table splitting; represents a performance evaluation factor under the i-th performance evaluation index; represents a total number of performance evaluation indexes; represents a correction factor corresponding to a number of n performance evaluation indexes selected;

[0195] and, ;

[0196] When the capacity optimization value is less than a first preset value and the performance value is less than a second preset value, a to-be-optimized index is determined, and the database connection pool parameters, the data storage structure, and the data query strategy are continuously optimized.

[0197] In this embodiment, the concurrent demand information refers to a number of concurrent requests that the system needs to process in the same period;

[0198] In this embodiment, the capacity optimization value refers to a numerical value used to represent the capacity optimization performance of the database connection pool before and after the database sharding and table splitting;

[0199] In this embodiment, the first data storage structure refers to the data storage structure before the database sharding and table splitting;

[0200] In this embodiment, the second data storage structure refers to the data storage structure after the database sharding and table splitting;

[0201] In this embodiment, the structural change data is change data obtained by comparing the first data storage structure with the second data storage structure;

[0202] In this embodiment, the running parameter change data is running parameter change data of the database connection pool before and after the process of splitting the database and the table;

[0203] In this embodiment, the first preset value is a preset parameter used for comparison with the capacity optimization value to determine whether the system needs to be optimized;

[0204] In this embodiment, the second preset value is a preset parameter used for comparison with the performance value to determine whether the system needs to be optimized, and is used in combination with the first preset value.

[0205] The implementation principle and beneficial effects of the present embodiment are as follows: by obtaining the concurrent demand information in the service demand information, the capacity optimization value of the database connection pool before and after the process of splitting the database and the table can be determined, and then the performance change data of the database connection pool is obtained by obtaining the data structure change information of the original database before and after the process of splitting the database and the table, so that the capacity optimization value and the performance value are compared with the preset parameters to determine the optimizable indexes of the database connection pool, the data storage structure and the data query class, so as to continuously improve the query performance and storage performance of the database.

[0206] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A dynamic database and table splitting method based on business requirements, characterized in that, The method comprises the following steps: Step 1: Obtain business requirement information and data information in an original database, perform feature extraction on the business requirement information and the data information, and generate a requirement-data feature set; Step 2: Obtain matched database and table splitting rules in a rule database based on the requirement-data feature set and in combination with a preset feature-rule matching table; Step 3: Allocate corresponding databases and table structures to original data and to-be-written data in the original database based on the database and table splitting rules, and establish a configuration relationship between the original data and the to-be-written data and the corresponding databases and table structures; Step 4: Split the original database based on the database and table splitting rules and the configuration relationship, and optimize the data storage structure and data query strategy after database and table splitting in combination with the business requirement information; In step 3, the method comprises the following steps: Obtain the database and table splitting rules and perform analysis to obtain database splitting rules, table splitting rules and database and table splitting mapping relationships; Obtain original data in the original database and corresponding data types and data structures, and output first data information; At the same time, obtain data types and performance requirements of to-be-written data corresponding to the business requirement information, and output second data information; Based on the first data information and the second data information, and in combination with a preset information-index matching table, obtain corresponding rule matching indexes; Based on the first data information, the second data information and the corresponding rule matching indexes, generate data-index allocation information; Based on the data-index allocation information, select corresponding allocation instructions in an instruction database; Based on the data-index allocation information and the allocation instructions, allocate corresponding database splitting rules and table splitting rules to the first data information and the second data information; At the same time, based on the data-index allocation information, and in combination with system selection instructions and manual selection instructions, select target database servers in a server candidate library; Based on the database splitting rules and the table splitting rules, output sub-database and sub-table allocation results under the target database servers; Based on the sub-database and sub-table allocation results, establish data-sub-table mapping relationships between the first data information, the second data information and the sub-database and sub-table; In combination with the database and table splitting mapping relationships and the data-sub-table mapping relationships, obtain configuration files under corresponding mapping relationships through a preset mapping-configuration matching module; Based on the configuration files, establish configuration relationships between the first data information, the second data information and the sub-database and sub-table.

2. The method of claim 1, wherein, In step 1, the method comprises the following steps: Obtain category requirements of a business and performance requirements under corresponding categories, generate business requirement information based on the category requirements and the corresponding performance requirements; The category requirements comprise data management requirements, business process requirements, cross-platform and integration requirements and user experience requirements.

3. The method of claim 1, wherein the method further comprises: In step 1, the method further comprises the following steps: Obtain category information, data structures and performance parameters of the original database, and comprehensively generate data information; The category information comprises a relational database, a non-relational database, a document database and a graph database. The data structure includes a table structure, an index, and a database management system structure.

4. The method of claim 1, wherein the method further comprises: In step 1, further comprising: Feature extraction is performed on the service requirement information to generate a first feature set, and feature extraction is performed on the data information to generate a second feature set; Feature matching is performed on the first feature set and the second feature set to output a requirement-data feature set.

5. The method of claim 1, wherein the method further comprises: In step 2, comprising: Selecting the features consistent with the requirement instructions from the requirement-data feature set to construct a to-be-matched feature set; Based on the to-be-matched feature set, corresponding rule matching information is selected from the pre-set feature-rule matching table; Based on the rule matching information, corresponding database and table division rules are selected from the rule database.

6. The method of claim 1, wherein the method further comprises: In step 4, comprising: Real-time monitoring is performed on the business requirement information, and business requirement acquisition logs are generated in combination with the time stamps corresponding to the business requirement information; Clustering analysis is performed on all business requirement information in a preset time period in the business requirement acquisition logs to output a requirement clustering result; Content analysis is performed on the business requirement information to obtain business first change information belonging to the first type in a preset time period; At the same time, business second change information belonging to the second type in a preset time period is obtained from the business requirement information; Statistical analysis is performed on the requirement clustering result, all business first change information and business second change information in a preset time period, in combination with the time stamp corresponding to the preset time period, to output business requirement change information; The latest configuration parameters of the database are obtained through real-time monitoring of the database connection pool by a preset monitoring module, and are output as first configuration information; Based on the business requirement change information and the database and table division rules, and in combination with the latest configuration parameters, configuration modification instructions are selected from the instruction database; Based on the configuration modification instructions, the configuration relationship corresponding to the business requirement change information is modified to generate configuration parameter modification information; Based on the configuration parameter modification information, configuration port information is obtained from the configuration-port correspondence table, and the port state corresponding to the configuration port information is switched to a modification state; The configuration parameters corresponding to the port in the modification state are modified, the first object corresponding to the original configuration parameters is deleted, and a second object is generated in combination with the first configuration information; The connection relationship between the second object and the first configuration information is established and the configuration relationship is updated; Based on the updated configuration relationship and in combination with the database and table division rules, the database and table division strategy is adjusted, and the data storage structure of the original database is updated based on the adjusted database and table division strategy.

7. The method of claim 1, wherein the method further comprises: In combination with the business requirement information, the data storage structure and the data query strategy after database and table division are optimized, including: According to the concurrent requirement information in the business requirement information, the capacity optimization value of the database connection pool is determined; ; wherein, represents a capacity optimization value; represents a maximum number of connections set to a database connection pool, represents a maximum number of connections determined based on the concurrency requirement information, represents a performance evaluation factor corresponding to the maximum number of connections; represents a reference value corresponding to ; represents a minimum number of connections set to a database connection pool; represents a minimum number of connections determined based on the concurrency requirement information; represents a performance evaluation factor corresponding to the minimum number of connections; represents a reference value corresponding to ; , represents a capacity incompatibility coefficient based on the maximum number of connections and the minimum number of connections; The first data storage structure before database and table division and the second data storage structure after database and table division are obtained, and the first data storage structure and the second data storage structure are compared to obtain structure change data; Based on the structure change data and in combination with the business demand information, running parameter change data of the database connection pool is obtained, and a performance value of the database connection pool is calculated; ; wherein, represents a performance value of the database connection pool; represents an exponential function; represents a ratio of the number of reused connections to the total number of connections in the database connection pool determined based on the running parameter change data; represents a ratio of the average value of the increase in the connection duration of all connections in the database connection pool determined based on the running parameter change data to the original average connection duration; represents the incremental connection ratio, represents a parameter value of the database connection pool under the i-th performance evaluation index before the database is sharded and the table is partitioned; represents a parameter value of the database connection pool under the i-th performance evaluation index after the database is sharded and the table is partitioned; represents a performance evaluation factor under the i-th performance evaluation index; represents the total number of performance evaluation indexes; represents a correction factor corresponding to the number n of selected performance evaluation indexes. and, ; when the capacity optimization value is less than the first preset value, and the performance value is less than the second preset value, determining the to-be-optimized index, and continuously optimizing the database connection pool parameter, the data storage structure and the data query strategy.

Images