Database partitioning method and apparatus

Abstract

The application provides a database split method and device, which can be applied to the technical field of big data. The method comprises the following steps: acquiring a split rule list, wherein the split rule in the split rule list is a split rule capable of acquiring data source information; intercepting a method for calling a database, reading business field information configured in an annotation used by the method; searching for a split rule from the split rule list based on the business field information; obtaining a shard key corresponding to the split rule; determining a data source identifier from the split rule based on the shard key; when the data source identifier is inconsistent with an identifier of a current data source, performing split database, and binding a data source corresponding to the determined data source identifier. The application can realize database split, has high efficiency, and is light in application.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a database partitioning method and device. Background Art

[0002] Database sharding is a common technique used in distributed databases. As data volumes continue to increase in the information age, traditional single database models face throughput bottlenecks. Database sharding can improve data storage efficiency and capacity. However, due to varying business needs, distributed databases are often used for different business scenarios, not just for full utilization, but only for a portion of the database. This requires the ability to control the scope of database usage while sharding.

[0003] The existing technical solution is to first define a global database partitioning rule and business-database range mapping in the configuration file, and then declare a custom algorithm in the database partitioning calculation algorithm. In this custom algorithm, it is first necessary to obtain the rule mapping and database range, and then perform matching calculations through SQL field values, and finally complete the database binding.

[0004] Taking values ​​from SQL fields will result in multiple rows being inserted into the database and multiple calculations being performed, reducing computational efficiency. Secondly, adding database range determination to the sharding algorithm binds and couples sharding calculations with database ranges, increasing the complexity of the sharding algorithm. Furthermore, the range determination logic is left to the user to implement throughout the entire process, making it less user-friendly. Summary of the Invention

[0005] The embodiment of the present invention provides a database partitioning method for implementing database partitioning with high efficiency and easy application. The method includes:

[0006] Obtain a database partitioning rule list, where the database partitioning rules in the database partitioning rule list are database partitioning rules that can obtain data source information;

[0007] Intercept the method that calls the database and read the business field information configured in the annotation used by the method;

[0008] Based on the business field information, searching for a sub-library rule from a sub-library rule list;

[0009] Obtain the sharding key corresponding to the sharding rule;

[0010] Based on the sharding key, determine the data source identifier from the sharding rule;

[0011] When the data source identifier is inconsistent with the identifier of the current data source, a database switch is performed, and the data source corresponding to the determined data source identifier is bound.

[0012] The embodiment of the present application provides a database splitting device, which is used for realizing database splitting, is high in efficiency and convenient in application, and comprises the following steps:

[0013] A database splitting rule module is configured to obtain a database splitting rule list, wherein the database splitting rules in the database splitting rule list are database splitting rules capable of obtaining data source information.

[0014] An annotation module is configured to intercept a method for calling a database, read business field information configured in an annotation used by the method, and find a database splitting rule from the database splitting rule list based on the business field information.

[0015] A database splitting calculation module is configured to obtain a shard key corresponding to the database splitting rule, determine a data source identifier from the database splitting rule based on the shard key, and bind a data source corresponding to the determined data source identifier when the data source identifier is inconsistent with an identifier of a current data source.

[0016] The embodiment of the present application further provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the database splitting method when executing the computer program.

[0017] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program for executing the database splitting method.

[0018] In the embodiment of the present application, the database splitting rule list is obtained, wherein the database splitting rules in the database splitting rule list are database splitting rules capable of obtaining data source information; the method for calling the database is intercepted, the business field information configured in the annotation used by the method is read; the database splitting rule is found from the database splitting rule list based on the business field information; the shard key corresponding to the database splitting rule is obtained; the data source identifier is determined from the database splitting rule based on the shard key; and the database is split when the data source identifier is inconsistent with the identifier of the current data source, and the determined data source identifier is bound to the data source. In the above process, different data sources are configured by using the database splitting rules, the database splitting use range can be dynamically changed according to the business, and the subsequent database splitting step is decoupled; the business field information configured in the annotation used by the method is read, so that the value can be obtained without modifying the original method for calling the database, and the overall efficiency is high. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. In the drawings:

[0020] Figure 1 Flow of the database splitting method in the embodiments of the present application Figure One ;

[0021] Figure 2 Flowchart of the process of obtaining the database splitting rule list in the embodiments of the present application

[0022] Figure 3 Principle diagram of the process of obtaining the database splitting rule list in the embodiments of the present application

[0023] Figure 4 Principle diagram of the process of binding the database splitting rule in the embodiments of the present application

[0024] Figure 5 Flowchart of the process of searching the database splitting rule in the embodiments of the present application

[0025] Figure 6 Principle diagram of the process of binding the data source in the embodiments of the present application

[0026] Figure 7 Flowchart of the process of determining the data source identifier in the embodiments of the present application

[0027] Figure 8 Flow of the database splitting method in the embodiments of the present application Figure Two ;

[0028] Figure 9 Schematic diagram of the database splitting device in the embodiments of the present application Figure One ;

[0029] Figure 10 Schematic diagram of the database splitting device in the embodiments of the present application Figure Two ;

[0030] Figure 11 Schematic diagram of the computer device in the embodiments of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will further describe the embodiments of the present application in combination with the drawings. Herein, the schematic embodiments of the present application and the descriptions thereof are used to explain the present application, but not as the limitation of the present application.

[0032] In the description of the present specification, "include", "includes", "have", "contains", and the like are open terms, that is, mean to include but not limited to. The description referring to the terms "one embodiment", "one specific embodiment", "some embodiments", "for example", and the like means that the specific features, structures, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Also, the specific features, structures, or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner. The order of the steps involved in each embodiment is used to illustrate the implementation of the present application, and the order of the steps is not limited. The order can be appropriately adjusted as needed.

[0033] Figure 1 The flow of the database splitting method in the embodiments of the present application Figure One As shown in Figure 1 , comprising:

[0034] Step 101, obtaining a database splitting rule list, wherein the database splitting rules in the database splitting rule list are database splitting rules capable of obtaining data source information;

[0035] Step 102, intercepting a method of calling a database, and reading business field information configured in an annotation used by the method;

[0036] Step 103, searching for a database splitting rule from the database splitting rule list based on the business field information;

[0037] Step 104, obtaining a shard key corresponding to the database splitting rule;

[0038] Step 105, determining a data source identifier from the database splitting rule based on the shard key;

[0039] Step 106, when the data source identifier is inconsistent with the identifier of the current data source, performing database switching, and binding the data source corresponding to the determined data source identifier.

[0040] In the embodiments of the present application, different data sources are configured using database splitting rules, the database splitting usage range can be dynamically changed according to business, and is decoupled from the subsequent database switching step; reading the business field information configured in the annotation used by the method enables the value to be obtained without modifying the original method of calling the database, and the overall efficiency is high.

[0041] In step 101, a database splitting rule list is obtained, wherein the database splitting rules in the database splitting rule list are database splitting rules capable of obtaining data source information.

[0042] Figure 2For the flowchart of acquiring the database splitting rule list in the embodiment of the present application, in an embodiment, the acquiring of the database splitting rule list comprises:

[0043] In step 201, the database splitting rules in the configuration file are loaded into the memory in the form of a file stream and encapsulated into objects.

[0044] In step 202, for each object corresponding to a database splitting rule, if data source information can be acquired from the object, the object is assembled into the database splitting rule list.

[0045] Figure 3 For the principle diagram of acquiring the database splitting rule list in the embodiment of the present application, and Figure 2 Correspondingly, step 201 is a parsing step. Step 202 contains a verification step and a screening step, wherein the acquisition of the data source information indicates that the database splitting rule is checked with the data source information in the existing data source module. If the data source information exists, it indicates that the database splitting rule is a useful database splitting rule, and the assembly step can be performed, that is, the useful database splitting rules screened out are assembled into the database splitting rule list and wait for binding.

[0046] Figure 4 For the principle diagram of the binding of the database splitting rule in the embodiment of the present application, it specifically comprises step 102 and step 103.

[0047] In step 102, the method of calling the database is intercepted, and the business field information configured in the annotation used by the method is read. The annotation is a self-defined annotation. In step 102, the interception corresponds to Figure 4 In the interception.

[0048] In an embodiment, the interception of the method of calling the database comprises:

[0049] The method of Dao layer using the self-defined annotation intercepts the method of calling the database through the Spring AOP mode.

[0050] In step 103, based on the business field information, the database splitting rule is searched from the database splitting rule list. There can be multiple methods, and one embodiment is given below.

[0051] Figure 5 For the flowchart of searching the database splitting rule in the embodiment of the present application, in an embodiment, based on the business field information, the database splitting rule is searched from the database splitting rule list, comprising:

[0052] In step 501, based on the business field information, the business field value in the argument of the method is acquired by using the reflection mode.

[0053] In step 502, the database splitting rule corresponding to the business field value in the database splitting rule list is searched.

[0054] wherein, step 501 and step 502 correspond to reflection and binding respectively. Figure 4

[0055] Figure 6 A schematic diagram for data source binding calculation in the embodiment of the application, and specifically comprising step 104-step 106.

[0056] In step 104, the shard key corresponding to the split database rule is obtained, for subsequent acquisition of the shard key key value.

[0057] In step 105, based on the shard key, the data source identifier is determined from the split database rule;

[0058] Figure 7 A flowchart for determining the data source identifier in the embodiment of the application, and in an embodiment, based on the shard key, the data source identifier is determined from the split database rule, comprising:

[0059] Step 701, based on the shard key, the shard key key value in the input parameter of the method is acquired by reflection;

[0060] Step 702, according to the shard key key value, the data source identifier is determined from the split database rule.

[0061] Step 701 corresponds to shard key value taking in Figure 6 , step 702 corresponds to data source calculation in Figure 6 , and step 106 corresponds to split database in Figure 6 .

[0062] In step 106, when the data source identifier is inconsistent with the identifier of the current data source, split database is performed, and the data source corresponding to the determined data source identifier is bound.

[0063] In an embodiment, split database is performed, comprising:

[0064] Split database is performed by using the split database method provided by JDBC.

[0065] Figure 8 A flowchart of the database split method in the embodiment of the application Figure Two , in an embodiment, the method further comprises:

[0066] Step 801, receiving a custom split database rule and a custom annotation for a method of calling a database; wherein, the split database rule is added with a data source identifier.

[0067] ​To sum up, in the method provided in the embodiment of the application, a database splitting rule list is acquired, the database splitting rules in the database splitting rule list are database splitting rules capable of acquiring data source information, a method of calling a database is intercepted, business field information configured in an annotation used by the method is read, a database splitting rule is searched from the database splitting rule list based on the business field information, a shard key corresponding to the database splitting rule is acquired, a data source identifier is determined from the database splitting rule based on the shard key, and when the data source identifier is inconsistent with an identifier of a current data source, database switching is performed, and a data source corresponding to the determined data source identifier is bound. In the above process, different data sources are configured using database splitting rules, the database splitting range can be dynamically changed according to business, and the subsequent database switching step is decoupled; the business field information configured in the annotation used by the method is read, so that the original method of calling a database does not need to be modified for value acquisition, and the overall efficiency is high. In addition, the database splitting rules are defined using a configuration file, which conforms to the development specification and makes the method and the device lightweight.

[0068] The application further provides a database splitting device, which has the same principle as the database splitting method and will not be described herein.

[0069] Figure 9 A schematic diagram of the database splitting device in the embodiment of the application Figure One As shown in Figure 9 , the device comprises:

[0070] A database splitting rule module 901 is configured to acquire a database splitting rule list, wherein the database splitting rules in the database splitting rule list are database splitting rules capable of acquiring data source information.

[0071] An annotation module 902 is configured to intercept a method of calling a database, read business field information configured in an annotation used by the method, and search a database splitting rule from the database splitting rule list based on the business field information.

[0072] A database splitting calculation module 903 is configured to acquire a shard key corresponding to the database splitting rule, determine a data source identifier from the database splitting rule based on the shard key, perform database switching when the data source identifier is inconsistent with an identifier of a current data source, and bind a data source corresponding to the determined data source identifier.

[0073] In an embodiment, the database splitting rule module is specifically configured to:

[0074] Load the database splitting rules in the configuration file into the memory in the form of a file stream and encapsulate the database splitting rules into objects;

[0075] For each object corresponding to a database splitting rule, if data source information can be acquired from the object, the object is assembled into the database splitting rule list.

[0076] In an embodiment, the annotation module is specifically configured to:

[0077] The Dao layer method using the custom annotation is intercepted by a Spring AOP method to call a database.

[0078] In an embodiment, the annotation module is specifically configured to:

[0079] Based on the business field information, the business field value in the input parameter of the method is obtained by using a reflection method;

[0080] The business field value corresponding to the database splitting rule in the database splitting rule list is searched.

[0081] In an embodiment, the database splitting calculation module is specifically configured to:

[0082] Based on the shard key, the shard key value in the input parameter of the method is obtained by using a reflection method;

[0083] According to the shard key value, the data source identifier is determined from the database splitting rule.

[0084] In an embodiment, the database splitting calculation module is specifically configured to:

[0085] The database splitting is performed by using a library switching method provided by JDBC.

[0086] Figure 10 A schematic diagram of the database splitting device in the embodiment of the application Figure Two As shown in Figure 10 The custom module 1001 is further included, configured to receive a custom database splitting rule and a custom annotation of a method for calling a database; and the database splitting rule is added with a data source identifier.

[0087] To sum up, in the method and device provided by the embodiment of the application, the database splitting rule module is used to obtain a database splitting rule list, and the database splitting rules in the database splitting rule list are database splitting rules that can obtain data source information; the annotation module is used to intercept a method of calling a database, read business field information configured in an annotation used by the method, find a database splitting rule from the database splitting rule list based on the business field information; the database splitting calculation module is used to obtain a shard key corresponding to the database splitting rule; determine a data source identifier from the database splitting rule based on the shard key; when the data source identifier is inconsistent with an identifier of a current data source, perform database switching, and bind a data source corresponding to the determined data source identifier. In the above process, different data sources are configured by using the database splitting rules, the database splitting range can be dynamically changed according to business, and the subsequent database switching step is decoupled; the business field information configured in the annotation used by the method is read, so that the original method of calling the database does not need to be modified for value taking, and the overall efficiency is high. In addition, the database splitting rules are defined by using a configuration file, which conforms to the development specification and makes the method and device lightweight.

[0088] The embodiment of the application further provides a computer device, Figure 11 The computer device can implement all steps in the database splitting method in the above embodiment, and specifically includes the following contents:

[0089] a processor 1101, a memory 1102, a communications interface 1103 and a communications bus 1104;

[0090] The processor 1101, the memory 1102 and the communications interface 1103 can communicate with each other through the communications bus 1104; the communications interface 1103 is used to realize information transmission between a server device, a detection device, a user terminal device and other related devices;

[0091] The processor 1101 is used to call a computer program in the memory 1102, and the processor implements all steps in the database splitting method in the above embodiment when executing the computer program.

[0092] The embodiment of the application further provides a computer readable storage medium, which can implement all steps in the database splitting method in the above embodiment, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement all steps in the database splitting method in the above embodiment.

[0093] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0094] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One means for carrying out each of the one or more functions specified in the flowchart and / or block diagram block or blocks.

[0095] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One means for carrying out each of the one or more functions specified in the flowchart and / or block diagram block or blocks.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One Figure One means for carrying out each of the one or more functions specified in the flowchart and / or block diagram block or blocks.

[0097] The specific embodiments described above have been disclosed by way of example and that, obviously, any modifications and / or alterations to the application shall be considered designed in the spirit and scope thereof. It is intended that the application be limited only by the claims.

Claims

1. A database partitioning method, characterized in that: include: Load the database partitioning rules in the configuration file into the memory in the form of a file stream and encapsulate them into an object; For each object corresponding to a database partitioning rule, if the data source information can be obtained from the object, the object is assembled into a database partitioning rule list. The database partitioning rules in the database partitioning rule list are database partitioning rules that can obtain the data source information. Intercept the method that calls the database and read the business field information configured in the annotation used by the method; Based on the business field information, searching for a database partitioning rule from a database partitioning rule list; Based on the business field information, searching for a sharding rule from a sharding rule list, including: obtaining a business field value in an input parameter of the method by using reflection based on the business field information; searching for a sharding rule corresponding to the business field value in the sharding rule list; Obtain the sharding key corresponding to the sharding rule; Based on the shard key, the shard key value in the input parameter of the method is obtained by reflection; According to the sharding key value, the data source identifier is determined from the sharding rules; When the data source identifier is inconsistent with the identifier of the current data source, a database switch is performed and the data source corresponding to the data source identifier is bound.

2. The database partitioning method according to claim 1, wherein: Intercept database calls, including: Use custom annotated Dao layer methods to intercept database calls through Spring AOP.

3. The database partitioning method according to claim 1, wherein: Perform library slicing, including: Use the database slicing method provided by JDBC to slicing the database.

4. The database partitioning method according to claim 1, wherein: Also includes: Receive custom database partitioning rules and custom annotations for a method of calling a database; wherein the database partitioning rules are added with a data source identifier.

5. A database sub-library device, characterized in that: include: The database partitioning rule module is used to load the database partitioning rules in the configuration file into the memory in the form of a file stream and encapsulate them into objects; For each object corresponding to a database partitioning rule, if the data source information can be obtained from the object, the object is assembled into a database partitioning rule list. The database partitioning rules in the database partitioning rule list are database partitioning rules that can obtain the data source information. The annotation module is used to intercept the method of calling the database and read the business field information configured in the annotation used by the method; Based on the business field information, searching for a database partitioning rule from a database partitioning rule list; Based on the business field information, searching for a sharding rule from a sharding rule list, including: obtaining a business field value in an input parameter of the method by using reflection based on the business field information; searching for a sharding rule corresponding to the business field value in the sharding rule list; The sharding calculation module is used to obtain the sharding key corresponding to the sharding rule; based on the sharding key, the sharding key value in the input parameter of the method is obtained by reflection; according to the sharding key value, the data source identifier is determined from the sharding rule; when the data source identifier is inconsistent with the identifier of the current data source, the database is cut and the data source corresponding to the data source identifier is bound.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 4.

Images