A method for implementing dynamic database partitioning

Abstract

The present application relates to a method for implementing dynamic sub-library, which belongs to the field of computer technology. The implementation method of the present application includes: parsing the form created by the system front end, extracting the business keywords in the form; querying the table-library relationship table in the main library, judging whether there is a sub-library corresponding to the business keyword, if so, performing the table building operation of the form based on the existing sub-library, otherwise creating a new library as the sub-library, and performing the table building operation of the form based on the newly created sub-library; and after performing the table building operation of the form, maintaining the mapping relationship between the form and the corresponding sub-library in the table-library relationship table. The present application can flexibly and conveniently complete the horizontal expansion of application business data, and to a certain extent solve the system risks brought about by application performance bottlenecks.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a method for implementing dynamic library partitioning. Background Art

[0002] With the development of computer and information technology, various information management systems have been widely used, greatly improving people's production and living efficiency. The implementation of such systems often involves database systems to manage business data in applications.

[0003] In reality, sometimes the form business types and data of an application (referring to a specific system applicable to a certain scenario) will show explosive growth, and the creation of business data tables is dynamically generated. The number of business database tables of an application may reach tens of thousands or even hundreds of thousands. If the data model of this application is built based on a single library of a relational database, system operation and maintenance will face huge challenges and risks.

[0004] For example, when the concurrent access volume of an application increases, since the number of connections between the single database and the application is a constant value, the connection between the application and the database will be exhausted, causing the database to crash and the application service to be unavailable. For another example, when it is necessary to perform data statistics on all tables under this application database based on a certain dimension, it is necessary to traverse all tables under this database, which will occupy more server resources (such as memory or CPU) for calculation, and the database service itself will have no resources to use, which will cause the application service to be unable to access the database service. For another example, the overall backup or archiving of the database will become extremely inefficient, not only facing the possibility of interruption of the backup or archiving process at any time, but also causing a huge waste of resources (for example, it may be necessary to only back up or archive the database tables of a certain business type at present), because it is difficult to perform selective data backup or archiving.

[0005] Therefore, when applications create business data tables in dynamic mode, how to avoid system risks caused by performance bottlenecks when business explodes has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to overcome the problems existing in the related technologies at least to a certain extent, the present application provides a method for implementing dynamic database partitioning, which specifically realizes the automatic adaptation connection between the system and multiple database data sources, and solves the technical problem of how to avoid the system risks caused by performance bottlenecks when the business explodes when the application creates business data tables in a dynamic mode.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] The present application provides a method for implementing dynamic library partitioning, which includes:

[0009] Parse the form created by the system front end and extract the business keywords in the form;

[0010] Query the table-library relationship table in the main database to determine whether there is a sub-library corresponding to the business keyword.

[0011] If it exists, create a table for the form based on the existing sub-library; otherwise, create a new library as the sub-library, and create a table for the form based on the newly created sub-library;

[0012] And after the table creation operation of the form is performed, the mapping relationship between the form and the corresponding sub-library is maintained in the table-library relationship table.

[0013] Optionally, for the existing sub-database, the table creation operation of the form specifically includes:

[0014] The data source connection channel between the adaptation system and the sub-library is adapted, and the table structure information corresponding to the form is created in the sub-library based on the adapted data source connection channel.

[0015] Optionally, for the newly created sub-database, the table creation operation of the form specifically includes:

[0016] A data source connection channel is created between the system and the sub-library, and table structure information corresponding to the form is created in the sub-library based on the created data source connection channel.

[0017] Optionally, the business keyword includes an identifier of the year in which the form was created, an identifier of a user organization that created the form, or an identifier of a client that created the form.

[0018] Optionally, in response to the user's operation on the business data of the target form at the system front end, a mapping relationship between the target form and its corresponding sub-library is searched from the main library;

[0019] Based on the found mapping relationship, a data source connection channel adapted to the corresponding sub-library is established, and the business data operation is performed based on the data source connection channel.

[0020] Optionally, the business data operation includes data submission, data query, data modification and data deletion.

[0021] Optionally, based on the Spring framework, the factory design pattern and the adapter design pattern are used to implement dynamic library partitioning.

[0022] This application adopts the above technical solution, which has at least the following beneficial effects:

[0023] In the technical solution of the present application, the form created by the system front end is parsed to extract the business keywords in the form; the table-library relationship table in the main database is queried to determine whether there is a sub-library corresponding to the business keyword. If so, the table creation operation of the form is performed based on the existing sub-library, otherwise a new database is created as the sub-library, and the table creation operation of the form is performed based on the newly created sub-library; and after the table creation operation of the form is performed, the mapping relationship between the form and the corresponding sub-library is maintained in the table-library relationship table. This implementation method of isolating the sub-library by the business keywords of the form and automatically expanding the database in the form of sub-library can flexibly and conveniently complete the horizontal expansion of application business data compared to a single database, and to a certain extent solves the system risks brought by application performance bottlenecks (such as the limitation of the number of connections of a single database).

[0024] Other advantages, objectives, and features of the present invention will be set forth in part in the following description, and in part will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide a further understanding of the technical solution of the present application or the prior art, and constitute a part of the specification. Among them, the accompanying drawings expressing the embodiments of the present application are used together with the embodiments of the present application to explain the technical solution of the present application, but do not constitute a limitation on the technical solution of the present application.

[0026] Figure 1 A flowchart of a method for implementing dynamic sub-library provided in one embodiment of the present application;

[0027] Figure 2 This is a flowchart schematically illustrating a process of creating a form library and a table in another embodiment of the present application;

[0028] Figure 3 A schematic diagram illustrating the process of implementing a business operation of a form in one embodiment of the present application;

[0029] Figure 4 This is a schematic diagram illustrating the algorithm logic for implementing dynamic library partitioning based on the Spring framework in one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other implementation methods obtained by ordinary technicians in the field without creative work belong to the scope of protection of the present application.

[0031] As described in the background technology, in practice, sometimes the form business types and data of an application (referring to a specific system applicable to a certain scenario) will show explosive growth, and the creation of business data tables is dynamically generated. The number of business database tables of an application may reach tens of thousands or even hundreds of thousands. If the data model of this application is built based on a single library of a relational database, system operation and maintenance will face huge challenges and risks.

[0032] For example, when the concurrent access volume of an application increases, since the number of connections between the single database and the application is a constant value, the connection between the application and the database will be exhausted, causing the database to crash and the application service to be unavailable. For another example, when it is necessary to perform data statistics on all tables under this application database based on a certain dimension, it is necessary to traverse all tables under this database, which will occupy more server resources (such as memory or CPU) for calculation, and the database service itself will have no resources to use, which will cause the application service to be unable to access the database service. For another example, the overall backup or archiving of the database will become extremely inefficient, not only facing the possibility of interruption of the backup or archiving process at any time, but also causing a huge waste of resources (for example, it may be necessary to only back up or archive the database tables of a certain business type at present), because it is difficult to perform selective data backup or archiving.

[0033] Therefore, when applications create business data tables in dynamic mode, how to avoid system risks caused by performance bottlenecks when business explodes has become a technical problem that needs to be solved urgently.

[0034] In view of this, in order to solve the problem of how to deal with the massive growth trend in the number of forms in the scenario where the application form model is dynamically generated, the present application provides a method for implementing dynamic database partitioning, which specifically implements a horizontally scalable business model, and also solves the performance bottleneck of the application's single database.

[0035] like Figure 1 As shown, in one embodiment, the implementation method of dynamic library partitioning proposed in this application includes:

[0036] Step S110, parsing the form created by the system front end to extract business keywords in the form;

[0037] In the present application, the business keyword is used for database (sub-database) isolation. Based on different scenarios, the business keywords here include the year identifier for creating the form, the user organization identifier for creating the form, or the client identifier for creating the form (such as different apps). For example, in this embodiment, the business keyword is the year identifier for creating the form. In this step, in actual implementation, the user creates a form from the application front end, submits and saves the form design, and the application back end parses the current form creation time and extracts the year value to obtain the business keyword of the form.

[0038] Then proceed to step S120, query the table-library relationship table in the main library (main database) to determine whether there is a sub-library corresponding to the business keyword. The main library in this application is relative to the sub-library. The sub-library is used to store and manage actual business data, and the main library is used to assist in the maintenance and management of the sub-library.

[0039] Then proceed to step S130. Here, continue with the previous example. If the extracted year value (business keyword) is 2021, and the sub-database corresponding to the year value (for example, DB_2021) already exists, then in step S130, the table creation operation of the form is performed based on the existing sub-database (that is, DB_2021).

[0040] If the submitted year value is a new year, such as 2023 (currently 2022), there is obviously no sub-library (DB_2023) corresponding to the year value. In this case, in step S130, a new library is created as the sub-library (i.e., the sub-library DB_2023 corresponding to the business keyword 2023), and the form creation operation is performed based on the newly created sub-library (DB_2023);

[0041] Here, the table creation operation in step S130 is described. For example, in step S130,

[0042] For the existing sub-database, a table creation operation of the form is performed, specifically including: adapting the data source connection channel between the system and the sub-database, and creating table structure information corresponding to the form in the sub-database based on the adapted data source connection channel;

[0043] For the newly created sub-database, a table creation operation of the form is performed, specifically including: creating a data source connection channel between the system and the sub-database, and creating table structure information corresponding to the form in the sub-database based on the created data source connection channel.

[0044] Then, in step S140, after the table creation operation of the form in step S130 is performed, the mapping relationship between the form and the corresponding sub-library is maintained in the table-library relationship table of the main library, and the creation of the new form is now completed.

[0045] In the technical solution of the present application, the form created by the system front end is parsed to extract the business keywords in the form; the table-library relationship table in the main database is queried to determine whether there is a sub-library corresponding to the business keyword. If so, the table creation operation of the form is performed based on the existing sub-library, otherwise a new database is created as the sub-library, and the table creation operation of the form is performed based on the newly created sub-library; and after the table creation operation of the form is performed, the mapping relationship between the form and the corresponding sub-library is maintained in the table-library relationship table. This implementation method of isolating the sub-library by the business keywords of the form and automatically expanding the database in the form of sub-library can flexibly and conveniently complete the horizontal expansion of application business data compared to a single database, and to a certain extent solves the system risks brought by application performance bottlenecks (such as the limitation of the number of connections of a single database).

[0046] To facilitate understanding of the technical solution of the present application, the technical solution of the present application is introduced and explained with reference to another embodiment below.

[0047] In this embodiment, the implementation method of the dynamic sub-library proposed in this application includes the creation of a form library (dynamic library), a table, and the business operation of the form, wherein the method flow of the creation of the form library and the table is as follows: Figure 2 As shown, it includes the following steps:

[0048] 1) The user creates a form from the application front end and submits it to save the form design;

[0049] 2) The application backend parses the current form creation time and extracts the year value;

[0050] 3) If the current year is a new year, determine whether the library corresponding to the year value exists;

[0051] If it does not exist, create a new database corresponding to the year value, such as DB_2023, and create an adaptive data source connection channel between the application and the new database. Create the table structure information corresponding to the form under the new database.

[0052] 4) Finally, the mapping relationship between the newly created library and the table is maintained in the table-library relationship table of the master library DB_master. At this point, the creation process of the new library and new table is completed;

[0053] If the library corresponding to the year value already exists, the adapter applies the data source connection channel corresponding to the year value of the current form creation, such as DB_2022, and creates the table structure information corresponding to the form in the corresponding year library.

[0054] Then, the same process is performed as in step 4) to maintain the mapping relationship between the table and the corresponding library in the table-library relationship table of the master library DB_master. At this point, the creation process of the new form is completed.

[0055] In this embodiment, the business operation of the form (i.e., the business operation part of the form in the implementation of dynamic sub-library) is implemented based on the routing library table technology. Figure 3 As shown, in this embodiment, the business operation of the form is implemented based on the routing library table technology, specifically including:

[0056] The user operates the form function from the front end of the application. For the business data operation of the target form by the user at the front end of the system (for example, the business data operation here includes data submission, data query, data modification and data deletion, etc.), the mapping relationship between the target form and its corresponding sub-database is found from the main database.

[0057] Then, based on the found mapping relationship, a data source connection channel adapted to the corresponding sub-database is established, and business data operations are performed based on the data source connection channel (ie, specific background operations corresponding to the front-end operations are performed).

[0058] Finally, the technical solution of this application is explained from the perspective of actual development and implementation. The dynamic database creation, table creation, dynamic routing library and table operations in the dynamic database implementation in this application are transparent and imperceptible to the front-end users of the application, but the back-end of the application requires a highly abstract, flexible and scalable design to ensure the horizontal development of the application business and the flexible expansion and stable performance of the application system.

[0059] In one embodiment, the present application is based on the Spring framework and uses the factory design pattern and the adapter design pattern in the software design pattern to implement dynamic library partitioning. Based on the design criteria mentioned above, by Figure 4 The algorithm design shown completes its implementation process, and its specific algorithm rules are as follows:

[0060] The first step is to start the Spring container. During the startup process, a dynamic data source management factory bean is created. This factory manages the automatic creation process of multiple data source beans, decoupling the creation and use of multiple data sources. That is, once created, they can be used by any business object. At the same time, it facilitates the unified creation, management, and maintenance of multiple data source objects.

[0061] In the second step, during the startup of the Spring container, after completing the first step, the Spring container reads the existing multiple data source information from the master database DB_Master, and dynamically creates a data source Bean object for each data source through the dynamic data source management factory Bean. When the object is initialized, the specific configuration information of the corresponding data source is loaded, including the data source name, database driver name, database access user name and password, and database connection pool parameter configuration information;

[0062] In the third step, the user creates a form and determines whether the year of the form creation time is a new year. If it is a new year, create a new library data source Bean; if it is still the current year, route to the corresponding data source through the dynamic data source adapter Adaptor and execute step 6;

[0063] The fourth step is to load the new library data source bean through the dynamic data source adapter Adaptor and establish a connection routing channel with the database;

[0064] Step 5: Execute the SQL statement for creating a new database through the data source connection channel established in step 4 to complete the creation of the new database.

[0065] Step 6: Execute the SQL statement to create a new table under the connection channel of the current library to complete the creation of the new table;

[0066] In the seventh step, the user performs business data operations on the existing form, such as adding, deleting, checking, and modifying. The dynamic data source adapter reads the data source corresponding to the modified form from the master database DB_Master, and then establishes a connection routing channel with the data source to complete the business data processing operation.

[0067] At this point, the entire dynamic database creation, table creation, and business operation dynamic routing algorithm rules have been completed.

[0068] This application uses the dynamic creation of form data models based on the design concept of application adaptation to multiple data sources. It automatically loads data sources and manages and adapts multiple data sources. For example, the year is used as the isolation condition for the application business library table to ensure the simplicity and reliability of the business library table creation logic. Then, through the multiple data source adapter, the front-end business operations can be automatically routed to adapt to the corresponding library tables on the back end, which can ensure that the operation of the application multiple data sources becomes flexible and transparent, and also ensures the flexible and horizontal expansion of the application business in multiple fields.

[0069] This application solves the problems of poor business scalability and serious system performance bottlenecks in the application's single library due to the growth of massive forms. By dynamically and automatically expanding the library and table, business operations are flexibly adapted to the corresponding library and table. On the one hand, the automatic horizontal expansion of the application business field becomes elastic and controllable, while also reducing the risk of possible operational errors caused by manual intervention; on the other hand, since the application expands the database connection channel in the form of multiple data sources, it provides a favorable optimization path for improving the performance bottleneck of high concurrent access to the application.

[0070] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person familiar with the technology within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for implementing dynamic sub-library, characterized in that: include: Parse the form created by the system front end and extract the business keywords in the form; Query the table-library relationship table in the main database to determine whether there is a sub-library corresponding to the business keyword. If it exists, create a table for the form based on the existing sub-library; otherwise, create a new library as the sub-library, and create a table for the form based on the newly created sub-library; And after the table creation operation of the form is performed, the mapping relationship between the form and the corresponding sub-library is maintained in the table-library relationship table; Among them, for the existing sub-library, the table creation operation of the form specifically includes: adapting the data source connection channel between the system and the sub-library, and creating the table structure information corresponding to the form in the sub-library based on the adapted data source connection channel; Wherein, for the newly created sub-database, the table creation operation of the form specifically includes: A data source connection channel is created between the system and the sub-library, and table structure information corresponding to the form is created in the sub-library based on the created data source connection channel; the business keywords include a year identifier for creating the form, a user organization identifier for creating the form, or a client identifier for creating the form.

2. The implementation method according to claim 1, characterized in that: In response to the user's operation on the business data of the target form at the system front end, the mapping relationship between the target form and its corresponding sub-database is searched from the main database; Based on the found mapping relationship, a data source connection channel adapted to the corresponding sub-library is established, and the business data operation is performed based on the data source connection channel.

3. The implementation method according to claim 2, characterized in that: The business data operations include data submission, data query, data modification and data deletion.

4. The implementation method according to any one of claims 1 to 3, characterized in that: Based on the Spring framework, the factory design pattern and the adapter design pattern are used to implement dynamic library partitioning.

Images