Database sub-library and sub-table expansion method, device and computer-readable storage medium

Abstract

The embodiment of the present application provides a method, device and computer-readable storage medium for expanding the capacity of database sub-libraries and sub-tables, including: obtaining the routing fields of each data table of the database, and constructing a first set of routing fields to be selected based on each routing field, and sending the first set of routing fields to be selected to each business end; when a business end generates data, storing each data in the corresponding data table based on the data shard ID, and the shard ID includes the routing field; when the amount of data stored in each data table in the database is not less than the first preset data amount, the database is expanded, and the routing fields of the new data table obtained by the expansion are added to the first set of routing fields to obtain a second set of routing fields to be selected, so that when each business end generates data, the data shard ID is generated based on the second set of routing fields to be selected. This solution makes it unnecessary to migrate the data stored before the expansion after the database is expanded, thereby reducing the operation and maintenance costs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a method and device for expanding the capacity of a database by sub-library and sub-table, and a computer-readable storage medium. Background Art

[0002] With the rapid development of business, the amount of data in many companies' databases will surge, and access performance will slow down, so optimization is imminent. Relational databases themselves are more likely to become system bottlenecks, and the storage capacity, number of connections, and processing power of a single machine are limited. Taking the MySQL database as an example, when the amount of data in a single table reaches 10 million or 100G, due to the large number of query dimensions, even if slaves are added and indexes are optimized, the performance will still deteriorate severely when performing many operations. At this time, it is necessary to use database splitting technology to solve performance problems.

[0003] The current database splitting process basically follows the order of vertical splitting, read-write separation, and horizontal splitting. Among them, horizontal splitting can be achieved through three methods: only splitting tables, only splitting libraries, and splitting libraries and tables. Splitting libraries and tables is a more commonly used implementation method. For example, for the database db (hereinafter referred to as the db library), split the library and table: split the db library into two libraries, db_0 and db_1, db_0 contains two sub-tables, user_0 and user_1, and db_1 contains two sub-tables, user_2 and user_3. Suppose there are 40 million data in the user table of the db library. Now split the db library into two sub-libraries, db_0 and db_1, and split the user table into four sub-tables, user_0, user_1, user_2, and user_3, each of which stores 10 million data.

[0004] After sharding the database and tables, it may be necessary to further expand the capacity of the shards. However, according to the current routing strategy for data storage in shards, expansion requires re-sharding of the shards before expansion, which requires downtime to migrate the data before expansion, resulting in high operation and maintenance costs. Summary of the invention

[0005] The purpose of this application is to solve at least one of the above technical defects. The technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for expanding the capacity of a database by sub-library and sub-table, including:

[0007] Obtaining a routing field composed of a sub-library identifier and a sub-table identifier of each data table currently in use in the database, building a first set of routing fields to be selected based on each routing field, and sending the first set of routing fields to be selected to each business terminal;

[0008] When a business end generates data, obtain the data's shard identity ID, and store each data in the corresponding data table based on the data's shard ID. The shard ID includes an identification field and a routing field. The routing field is obtained from the first set of candidate routing fields by the business end corresponding to each data generation.

[0009] When the amount of data stored in each data table in the database is not less than the first preset data amount, the database is expanded by sub-libraries and tables, and the routing field obtained by combining the sub-library identifier and the sub-table identifier of the new data table obtained by the expansion is added to the first set of candidate routing fields to obtain a second set of candidate routing fields, and the second set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the second set of candidate routing fields.

[0010] In an optional embodiment of the present application, a first set of to-be-selected routing fields is constructed based on each routing field, including:

[0011] The routing fields corresponding to the data tables are added to the first set of routing fields to be selected in equal proportion, so that the number of routing fields in the first set of routing fields to be selected is equal.

[0012] In an optional embodiment of the present application, the identification field further includes a time field, an Internet Protocol address IP field, and a sequence field;

[0013] Among them, the time field is used to indicate the generation time of the data, the IP field is used to indicate the IP of the service end that generates the data, and the sequence field is used to indicate the sequence number of the data under the same generation time and IP.

[0014] In an optional embodiment of the present application, storing each data in a corresponding data table based on the shard ID of the data includes:

[0015] Based on the routing field in the data shard ID, obtain the corresponding shard database ID and shard table ID;

[0016] Based on the sub-library identifier and the sub-table identifier, the data table where the data is to be stored is determined, and the identification field of the data is stored in the data table where the data is to be stored as the primary key value of the data corresponding to the data.

[0017] In an optional embodiment of the present application, a routing field obtained by combining a sub-library identifier and a sub-table identifier of a new data table obtained by expansion is added to the first set of candidate routing fields to obtain a second set of candidate routing fields, including:

[0018] After capacity expansion, obtain the remaining capacity of all data tables in the database;

[0019] Based on the remaining capacity of each data table, the routing fields corresponding to the new data table are added to the first set of candidate routing fields, so that the ratio of the number of routing fields in the second set of candidate routing fields is equal to the ratio of the remaining capacity of the data table corresponding to each routing field.

[0020] In an optional embodiment of the present application, the method further includes:

[0021] After the expansion, when the amount of data stored in the first target data table among all new data tables in the database is not less than the second preset amount of data, the number of each routing field in the second set of candidate routing fields is adjusted so that the proportion of the number of routing fields corresponding to the first target data table in the obtained third set of candidate routing fields is reduced, and the third set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the third set of candidate routing fields;

[0022] The second preset data amount is smaller than the first preset data amount.

[0023] In an optional embodiment of the present application, the method further includes:

[0024] After the capacity expansion, when the amount of data stored in the second target data table among all the data tables of the database is not less than the third preset amount of data, the routing field corresponding to the second target data table is deleted from the second set of candidate routing fields to obtain a fourth set of candidate routing fields, and the fourth set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the fourth set of candidate routing fields;

[0025] The third preset data volume is greater than the first preset data volume.

[0026] In a second aspect, an embodiment of the present application provides a database sub-library and sub-table expansion device, comprising:

[0027] A first candidate routing field set acquisition module is used to acquire routing fields composed of sub-library identifiers and sub-table identifiers of each data table currently in use in the database, and to construct a first candidate routing field set based on each routing field, and to send the first candidate routing field set to each business terminal;

[0028] A data storage module, used for obtaining a shard identity ID of the data when a business end generates data, and storing each data in a corresponding data table based on the shard ID of the data, wherein the shard ID includes an identification field and a routing field, and the routing field is obtained from the first set of candidate routing fields by the business end corresponding to each data generation;

[0029] The second candidate routing field set acquisition module is used to expand the database by sub-libraries and sub-tables when the amount of data stored in each data table in the database is not less than the first preset amount, and add the routing field obtained by combining the sub-library identifier and the sub-table identifier of the new data table obtained by the expansion to the first candidate routing field set to obtain the second candidate routing field set, and send the second candidate routing field set to each business terminal, so that when each business terminal generates data, it generates a data shard ID based on the second candidate routing field set.

[0030] In a first optional embodiment of the present application, the first candidate route field set acquisition module is specifically used for:

[0031] The routing fields corresponding to the data tables are added to the first set of routing fields to be selected in equal proportion, so that the number of routing fields in the first set of routing fields to be selected is equal.

[0032] In an optional embodiment of the present application, the identification field further includes a time field, an Internet Protocol address IP field, and a sequence field;

[0033] Among them, the time field is used to indicate the generation time of the data, the IP field is used to indicate the IP of the service end that generates the data, and the sequence field is used to indicate the sequence number of the data under the same generation time and IP.

[0034] In an optional embodiment of the present application, the data storage module is specifically used for:

[0035] Based on the routing field in the data shard ID, obtain the corresponding shard database ID and shard table ID;

[0036] Based on the sub-library identifier and the sub-table identifier, the data table where the data is to be stored is determined, and the identification field of the data is stored in the data table where the data is to be stored as the primary key value of the data corresponding to the data.

[0037] In an optional embodiment of the present application, the second candidate route field set acquisition module is specifically used for:

[0038] After capacity expansion, obtain the remaining capacity of all data tables in the database;

[0039] Based on the remaining capacity of each data table, the routing fields corresponding to the new data table are added to the first set of candidate routing fields, so that the ratio of the number of routing fields in the second set of candidate routing fields is equal to the ratio of the remaining capacity of the data table corresponding to each routing field.

[0040] In an optional embodiment of the present application, the device further includes a first adjustment module, which is used to:

[0041] After the expansion, when the amount of data stored in the first target data table among all new data tables in the database is not less than the second preset amount of data, the number of each routing field in the second set of candidate routing fields is adjusted so that the proportion of the number of routing fields corresponding to the first target data table in the obtained third set of candidate routing fields is reduced, and the third set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the third set of candidate routing fields;

[0042] The second preset data amount is smaller than the first preset data amount.

[0043] In an optional embodiment of the present application, the device further includes a second adjustment module, which is used to:

[0044] After the capacity expansion, when the amount of data stored in the second target data table among all the data tables of the database is not less than the third preset amount of data, the routing field corresponding to the second target data table is deleted from the second set of candidate routing fields to obtain a fourth set of candidate routing fields, and the fourth set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the fourth set of candidate routing fields;

[0045] The third preset data volume is greater than the first preset data volume.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor;

[0047] A computer program is stored in the memory;

[0048] A processor is used to execute a computer program to implement the method provided in the embodiment of the first aspect or any optional embodiment of the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the embodiment of the first aspect or any optional embodiment of the first aspect is implemented.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that when the computer device executes, the method provided in the embodiment of the first aspect or any optional embodiment of the first aspect is implemented.

[0051] The beneficial effects of the technical solution provided by this application are:

[0052] When the business end generates data, it sets a shard ID containing a routing field for the data, and the routing field is selected from the routing field set provided by the middleman. Each routing field in the routing field set indicates a shard table in the expanded database. Therefore, after the expansion, the middleware can directly store the data in the required storage data table according to the routing ID in the shard ID of the data generated by each business end, so that after the database is expanded, there is no need to migrate the data stored before the expansion, reducing the operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0054] Figure 1 An architecture diagram of the system on which the database sharding and table expansion solution provided in the embodiment of the present application depends;

[0055] Figure 2 A schematic diagram of a process for expanding the capacity of a database by dividing the database and the tables provided in an embodiment of the present application;

[0056] Figure 3 A structural block diagram of a database sub-library and sub-table expansion device is provided for an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0059] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0060] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0061] Figure 1 The architecture diagram of the system on which the database sharding and table expansion solution provided in the embodiment of the present application relies, the system may include multiple business terminals 101, middleware 102 and database 103, wherein the data generated by the business terminal 101 is parsed by the middleware 102 and stored in the database 103, and the middleware 102 may also configure the database 103, for example, the middleware 102 may perform sharding and table expansion operations on the database 103. After the sharding and table expansion processing, the database 103 may generally contain multiple shards, and each shard may contain multiple shards. The middleware 102 needs to parse the sharding ID of the data generated by the business terminal 101, and then route it to the corresponding sharding and table in the database for storage according to the indication of the sharding ID. Based on this, the database sharding and table expansion solution provided in the embodiment of the present application will be described in detail below.

[0062] Figure 2 A schematic diagram of a method for expanding the capacity of a database by dividing the database and the table provided in the embodiment of the present application. The execution subject of the method can be Figure 1 The intermediates shown, such as Figure 2 As shown, the method may include:

[0063] Step S201, obtain the routing fields composed of the sub-library identifier and the sub-table identifier of each data table currently in use in the database, build a first set of routing fields to be selected based on each routing field, and send the first set of routing fields to be selected to each business terminal.

[0064] Among them, the data table currently being used in the database is a data table that can store data generated by the business end.

[0065] Among them, the business end can be various applications (APPs), such as shopping APPs, and the order data generated by them is the data that needs to be stored in the embodiment of the present application.

[0066] Specifically, first, the intermediate monitors the data tables in the database to determine which data tables can continue to store data. Then, the intermediate obtains the identifiers of the sub-libraries where these data tables are located, activates the sub-library identifier of the zone data table, and then obtains the sub-table identifiers of these data tables, and combines the sub-library identifier and sub-table identifier of each data table, and obtains the routing field corresponding to the data table. For example, there are two sub-tables db_00 and db_01 in the database, and the sub-library identifiers are 00 and 01 respectively, and db_00 contains two data tables tb_00 and tb_01, and the sub-table identifiers are t00 and 01 respectively, then the routing field corresponding to the sub-table tb_00 is 0000, and the routing field corresponding to the sub-table tb_01 is 0001.

[0067] It can be understood that a data table can be uniquely identified from the database based on the routing field of a data table, and the shard ID of the data on the business end contains the routing field corresponding to the data table to be stored. Therefore, the middleware can find the data table to be stored based on the routing field in the shard ID of the data, thereby completing the data storage.

[0068] Furthermore, in order to ensure that the data generated by the business end can contain valid routing fields, the middleware will send the first set of candidate routing fields consisting of the routing fields corresponding to the data tables in use in the database to each business end, so that each business end can select the routing fields corresponding to the data table to be stored from the first set of candidate routing fields after generating data. Specifically, the middleware can send the first set of candidate routing fields to Zookeeper, and then notify each business end through Zookeeper.

[0069] Step S202, when a business end generates data, obtain the data's shard identity ID, and store each data in the corresponding data table based on the data's shard ID. The shard ID includes an identification field and a routing field. The routing field is obtained from the first set of candidate routing fields for the business end corresponding to each data generation.

[0070] Specifically, as described in the previous step, the middleware sends the first set of candidate routing fields to each business end in advance. When each business end has data that needs to generate a shard ID for the data, in addition to generating an identification field according to a predetermined rule, it is also necessary to select its routing field from the first set of candidate routing fields. The identification field of the data can distinguish the data from other data. The middleware can store each data in the corresponding data table according to the shard ID of each data.

[0071] Step S203, when the amount of data stored in each data table in the database is not less than the first preset data amount, the database is expanded with sub-libraries and sub-tables, and the routing field obtained by combining the sub-library identifier and the sub-table identifier of the new data table obtained by the expansion is added to the first set of candidate routing fields to obtain a second set of candidate routing fields, and the second set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the second set of candidate routing fields.

[0072] Specifically, when the amount of data stored in each data table in the database is not less than the first preset data amount, the capacity of each data table in the database is no longer sufficient, so more capacity needs to be obtained through capacity expansion. After the middleware adds sub-libraries and sub-tables through sub-libraries and sub-tables, it constructs the routing fields corresponding to each database currently in use in the expanded database as a second set of routing fields to be selected. The middleware then sends the second set of routing fields to be selected to each business end, so that each business end can select the routing field corresponding to the data table to be stored from the second set of routing fields to be selected after generating data. Specifically, the middleware can send the first set of routing fields to be selected to Zookeeper, and then notify each business end through Zookeeper.

[0073] For example, before the expansion, the database contains two sub-databases db_00 and db_01, and each sub-database has 10 sub-tables tb_00~tb_09. Then, the first two digits of the routing field of the data generated by the business end before the expansion are between 00 and 01, and the last two digits are always between 00 and 09. When it is necessary to expand to 10 sub-databases, each with 10 sub-tables. Then the intermediate first creates sub-databases db_02~db_09 in the database, and then creates 10 sub-tables tb_00~tb_09 in each sub-database. Then, the second set of candidate routing fields obtained after the expansion is sent to each business end.

[0074] It can be understood that after the expansion, the intermediary sends the second routing field set consisting of the routing fields corresponding to all the data tables in use in the database to each business end. When generating data, each business end selects the routing field from the second routing field set to generate the shard ID of the data. When the intermediary stores the data generated by each business end into the database, it can directly find the data table to be stored according to the routing field in the shard ID of the data, that is, there is no need to migrate the original stored data after the database is expanded through sharding.

[0075] The solution provided by the present application is that when the business end generates data, a shard ID containing a routing field is set for the data, and the routing field is selected from a routing field set provided by the middleman, and each routing field in the routing field set indicates a shard table in the database after expansion. Therefore, after expansion, the middleware can directly store the data in the required storage data table according to the routing ID in the shard ID of the data generated by each business end, so that after the database is expanded, there is no need to migrate the data stored before the expansion, thereby reducing the operation and maintenance costs.

[0076] In an optional embodiment of the present application, a first set of to-be-selected routing fields is constructed based on each routing field, including:

[0077] The routing fields corresponding to the data tables are added to the first set of routing fields to be selected in equal proportion, so that the number of routing fields in the first set of routing fields to be selected is equal.

[0078] Specifically, the routing fields corresponding to each data table are added to the first set of candidate routing fields in equal proportion so that the number of routing fields in the first set of candidate routing fields is equal. This is to ensure that when each business end generates a shard ID for the generated data, the probability of selecting each routing field in the first set of candidate routing fields is equal, thereby ensuring that the data generated by each business system can be evenly stored in each data table in the database to avoid data hot spots.

[0079] In an optional embodiment of the present application, the identification field further includes a time field, an Internet Protocol address IP field, and a sequence field;

[0080] Among them, the time field is used to indicate the generation time of the data, the IP field is used to indicate the IP of the service end that generates the data, and the sequence field is used to indicate the sequence number of the data under the same generation time and IP.

[0081] For example, for a certain data, the shard ID generated by the corresponding service end may include 32 bits, 12 bits of time field + 10 bits of IP field + 6 bits of sequence field + 4 bits of routing field, as follows:

[0082] 12-bit time field: The format is yyMMddHHmmss (each two digits corresponds to year, month, day, hour, minute, and second). The time field can be placed at the front of the shard ID to ensure that the shard ID trend of the data increases.

[0083] 10-bit IP field: Convert the 12-bit IP of each service end into a decimal number to obtain the IP field.

[0084] 6-bit sequence field: used to indicate the sequence number (0 to 999999) of data at the same generation time and IP.

[0085] 4-bit routing field: also known as database expansion bit, in order to achieve dynamic expansion without migrating data, 2 bits represent sub-library ID, 2 bits represent sub-table ID, and can be expanded to a maximum of 10,000 tables. Assuming that each table stores 10 million data, a total of 100 billion data can be stored. Of course, the number of bits in the routing field can be set according to demand, for example, it can be set to 6 bits, the first three bits identify the sub-library ID, and the last three bits identify the sub-table ID.

[0086] In an optional embodiment of the present application, storing each data in a corresponding data table based on the shard ID of the data includes:

[0087] Based on the routing field in the data shard ID, obtain the corresponding shard database ID and shard table ID;

[0088] Based on the sub-library identifier and the sub-table identifier, the data table where the data is to be stored is determined, and the identification field of the data is stored in the data table where the data is to be stored as the primary key value of the data corresponding to the data.

[0089] After the expansion, since there is no data stored in the newly added data table, and the previous data table has stored a lot of data, it is hoped that more data generated by each business end will be stored in the newly added data table. To achieve this goal, it is necessary to adjust the proportion of each routing field in the second candidate routing field set.

[0090] Then, in an optional embodiment of the present application, a routing field obtained by combining a sub-library identifier and a sub-table identifier of a new data table obtained by expansion is added to the first set of candidate routing fields to obtain a second set of candidate routing fields, including:

[0091] After capacity expansion, obtain the remaining capacity of all data tables in the database;

[0092] Based on the remaining capacity of each data table, the routing fields corresponding to the new data table are added to the first set of candidate routing fields, so that the ratio of the number of routing fields in the second set of candidate routing fields is equal to the ratio of the remaining capacity of the data table corresponding to each routing field.

[0093] For example, assuming that before capacity expansion, the first set of candidate routing fields is {0000, 0001}, and during capacity expansion, the remaining capacity of the data tables corresponding to the two routing fields "0000" and "0001" is 40% of the original capacity, and after capacity expansion, a sub-library is added, and there is a sub-table in the sub-library. Then, according to the above configuration, the ratio of the remaining capacity of the existing data table before capacity expansion to the newly added data table is 2:2:5. Therefore, the second set of candidate routing fields can be {0000, 0000, 0001, 0001, 0100, 0100, 0100, 0100, 0100}.

[0094] In an optional embodiment of the present application, the method may further include:

[0095] After the expansion, when the amount of data stored in the first target data table among all new data tables in the database is not less than the second preset amount of data, the number of each routing field in the second set of candidate routing fields is adjusted so that the proportion of the number of routing fields corresponding to the first target data table in the obtained third set of candidate routing fields is reduced, and the third set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the third set of candidate routing fields;

[0096] The second preset data amount is smaller than the first preset data amount.

[0097] Specifically, after the expansion, in the process of continuous data storage, more and more data is stored in the newly added data table. If the amount of data in a newly added data table reaches the second preset data amount, the probability of storing data in the newly added data table can be reduced, that is, the probability of each business end selecting the routing field corresponding to the data table when generating the shard ID of the data can be reduced. Then, the number of routing fields corresponding to the data table in the second set of routing fields to be selected can be reduced, so that the proportion of the number of routing fields corresponding to the first target data table in the obtained third set of routing fields to be selected is reduced.

[0098] For example, the second set of candidate routing fields obtained after expansion is {0000, 0000, 0001, 0001, 0100, 0100, 0100, 0100, 0100}. During the data storage process, the data volume of the data table corresponding to "0100" reaches the second preset data volume. Then, the routing field can be reduced to obtain the third routing field set {0000, 0000, 0001, 0001, 0100, 0100, 0100}.

[0099] It should be noted that the first preset data amount and the second data amount mentioned above can be set according to actual needs, and the second preset data amount must be smaller than the first preset data amount.

[0100] In an optional embodiment of the present application, the method may further include:

[0101] After the capacity expansion, when the amount of data stored in the second target data table among all the data tables of the database is not less than the third preset amount of data, the routing field corresponding to the second target data table is deleted from the second set of candidate routing fields to obtain a fourth set of candidate routing fields, and the fourth set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the fourth set of candidate routing fields;

[0102] The third preset data volume is greater than the first preset data volume.

[0103] Specifically, after the expansion, in the process of continuously storing data, the amount of data in some data tables in the database approaches the limit. In order to reserve a certain amount of fault tolerance, the data tables are generally not filled. Then, when the amount of data in a certain data table reaches the third preset amount of data, no more data can be stored in the data table. Therefore, the routing field corresponding to the data table can be deleted from the second set of routing fields to be selected.

[0104] For example, the second set of candidate routing fields obtained after expansion is {0000, 0000, 0001, 0001, 0100, 0100, 0100, 0100, 0100}. During the data storage process, the data volume of the data table corresponding to "0000" reaches the third preset data volume. Then, the routing field can be deleted to obtain the third routing field set {0001, 0001, 0100, 0100, 0100, 0100, 0100}.

[0105] It should be noted that the first preset data amount and the second data amount mentioned above can be set according to actual needs, and the third preset data amount must be greater than the first preset data amount.

[0106] It is understandable that each time each service end generates data for routing field selection, a routing field is randomly selected from a complete set of routing fields to be selected (the first set of routing fields to be selected, the second set of routing fields to be selected, or the third set of routing fields to be selected). For example, when service end A generates data 1, a routing field is randomly selected from the second set of routing fields to be selected {0000, 0000, 0001, 0001, 0100, 0100, 0100, 0100, 0100} to generate the shard ID of data 1. When service end A generates data 2 immediately afterwards, the middleware does not send a new set of routing fields to be selected, so a routing field is randomly selected from the second set of routing fields to be selected {0000, 0000, 0001, 0001, 0100, 0100, 0100, 0100, 0100} to generate the shard ID of data 2.

[0107] Figure 3 The present application provides a structural block diagram of a database sub-library and sub-table expansion device, such as Figure 3 As shown, the device 300 may include: a first to-be-selected routing field set acquisition module 301, a data storage module 302, and a second to-be-selected routing field set acquisition module, wherein:

[0108] The first candidate routing field set acquisition module 301 is used to acquire routing fields composed of sub-library identifiers and sub-table identifiers of each data table currently being used in the database, and to construct a first candidate routing field set based on each routing field, and to send the first candidate routing field set to each service end;

[0109] The data storage module 302 is used to obtain the data shard identity ID when a business end generates data, and store each data in a corresponding data table based on the data shard ID. The shard ID includes an identification field and a routing field. The routing field is obtained from the first set of candidate routing fields by the business end corresponding to each data generation;

[0110] The second candidate routing field set acquisition module 303 is used to expand the database by sub-libraries and tables when the amount of data stored in each data table in the database is not less than the first preset amount, and add the routing field obtained by combining the sub-library identifier and the sub-table identifier of the new data table obtained by the expansion to the first candidate routing field set to obtain the second candidate routing field set, and send the second candidate routing field set to each business terminal, so that when each business terminal generates data, it generates a data shard ID based on the second candidate routing field set.

[0111] The solution provided by the present application is that when the business end generates data, a shard ID containing a routing field is set for the data, and the routing field is selected from a routing field set provided by the middleman, and each routing field in the routing field set indicates a shard table in the database after expansion. Therefore, after expansion, the middleware can directly store the data in the required storage data table according to the routing ID in the shard ID of the data generated by each business end, so that after the database is expanded, there is no need to migrate the data stored before the expansion, thereby reducing the operation and maintenance costs.

[0112] In a first optional embodiment of the present application, the first candidate route field set acquisition module is specifically used for:

[0113] The routing fields corresponding to the data tables are added to the first set of routing fields to be selected in equal proportion, so that the number of routing fields in the first set of routing fields to be selected is equal.

[0114] In an optional embodiment of the present application, the identification field further includes a time field, an Internet Protocol address IP field, and a sequence field;

[0115] Among them, the time field is used to indicate the generation time of the data, the IP field is used to indicate the IP of the service end that generates the data, and the sequence field is used to indicate the sequence number of the data under the same generation time and IP.

[0116] In an optional embodiment of the present application, the data storage module is specifically used for:

[0117] Based on the routing field in the data shard ID, obtain the corresponding shard database ID and shard table ID;

[0118] Based on the sub-library identifier and the sub-table identifier, the data table where the data is to be stored is determined, and the identification field of the data is stored in the data table where the data is to be stored as the primary key value of the data corresponding to the data.

[0119] In an optional embodiment of the present application, the second candidate route field set acquisition module is specifically used for:

[0120] After capacity expansion, obtain the remaining capacity of all data tables in the database;

[0121] Based on the remaining capacity of each data table, the routing fields corresponding to the new data table are added to the first set of candidate routing fields, so that the ratio of the number of routing fields in the second set of candidate routing fields is equal to the ratio of the remaining capacity of the data table corresponding to each routing field.

[0122] In an optional embodiment of the present application, the device further includes a first adjustment module, which is used to:

[0123] After the expansion, when the amount of data stored in the first target data table among all new data tables in the database is not less than the second preset amount of data, the number of each routing field in the second set of candidate routing fields is adjusted so that the proportion of the number of routing fields corresponding to the first target data table in the obtained third set of candidate routing fields is reduced, and the third set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the third set of candidate routing fields;

[0124] The second preset data amount is smaller than the first preset data amount.

[0125] In an optional embodiment of the present application, the device further includes a second adjustment module, which is used to:

[0126] After the capacity expansion, when the amount of data stored in the second target data table among all the data tables of the database is not less than the third preset amount of data, the routing field corresponding to the second target data table is deleted from the second set of candidate routing fields to obtain a fourth set of candidate routing fields, and the fourth set of candidate routing fields is sent to each business end, so that when each business end generates data, a shard ID of the data is generated based on the fourth set of candidate routing fields;

[0127] The third preset data volume is greater than the first preset data volume.

[0128] Reference below Figure 4 , which shows an electronic device suitable for implementing the embodiments of the present application (for example, executing Figure 2 The electronic device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] The electronic device includes: a memory and a processor, the memory is used to store a program for executing the method described in each of the above method embodiments; the processor is configured to execute the program stored in the memory. The processor here can be referred to as the processing device 401 described below, and the memory can include at least one of the read-only memory (ROM) 402, the random access memory (RAM) 403, and the storage device 408 described below, as shown below:

[0130] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0131] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0132] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0133] It should be noted that the computer-readable storage medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0134] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0135] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0136] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0137] A routing field formed by combining a sub-library identifier and a sub-table identifier of each data table currently in use in the database is obtained, and a first set of routing fields to be selected is constructed based on each routing field, and the first set of routing fields to be selected is sent to each business terminal; when a business terminal generates data, a shard identity ID of the data is obtained, and each data is stored in a corresponding data table based on the shard ID of the data, the shard ID includes an identification field and a routing field, and the routing field is obtained from the first set of routing fields to be selected by the corresponding business terminal when each data is generated; when the amount of data stored in each data table in the database is not less than a first preset amount of data, the database is expanded by sub-libraries and sub-tables, and a routing field obtained by combining a sub-library identifier and a sub-table identifier of a new data table obtained by the expansion is added to the first set of routing fields to obtain a second set of routing fields to be selected, and the second set of routing fields to be selected is sent to each business terminal, so that when each business terminal generates data, a shard ID of the data is generated based on the second set of routing fields to be selected.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The modules or units involved in the embodiments of the present application may be implemented by software or hardware. The name of a module or unit does not limit the unit itself in some cases. For example, a proxy link acquisition module may also be described as a "module for acquiring proxy links".

[0141] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0142] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific method implemented when the above-described computer-readable medium is executed by an electronic device can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0144] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that when the computer device executes the computer instructions, the following conditions are achieved:

[0145] A routing field formed by combining a sub-library identifier and a sub-table identifier of each data table currently in use in the database is obtained, and a first set of routing fields to be selected is constructed based on each routing field, and the first set of routing fields to be selected is sent to each business terminal; when a business terminal generates data, a shard identity ID of the data is obtained, and each data is stored in a corresponding data table based on the shard ID of the data, the shard ID includes an identification field and a routing field, and the routing field is obtained from the first set of routing fields to be selected by the corresponding business terminal when each data is generated; when the amount of data stored in each data table in the database is not less than a first preset amount of data, the database is expanded by sub-libraries and sub-tables, and a routing field obtained by combining a sub-library identifier and a sub-table identifier of a new data table obtained by the expansion is added to the first set of routing fields to obtain a second set of routing fields to be selected, and the second set of routing fields to be selected is sent to each business terminal, so that when each business terminal generates data, a shard ID of the data is generated based on the second set of routing fields to be selected.

[0146] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0147] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for expanding the capacity of a database by dividing the database and the table. It is characterized in that include: Obtaining a routing field composed of a sub-library identifier and a sub-table identifier of each data table currently in use in the database, building a first set of routing fields to be selected based on each routing field, and sending the first set of routing fields to be selected to each service end; When a business end generates data, obtain the shard identity ID of the data, and store each data in a corresponding data table based on the shard identity ID of the data, wherein the shard identity ID includes an identification field and a routing field, and the routing field is obtained from the first set of candidate routing fields by the business end corresponding to each data generation; When the amount of data stored in each data table in the database is not less than the first preset amount of data, the database is expanded by sub-libraries and sub-tables, and a routing field obtained by combining a sub-library identifier and a sub-table identifier of a new data table obtained by the expansion is added to the first set of candidate routing fields to obtain a second set of candidate routing fields, and the second set of candidate routing fields is sent to each business terminal, so that when each business terminal generates data, a shard identity ID of the data is generated based on the second set of candidate routing fields; The routing field obtained by combining the sub-library identifier and the sub-table identifier of the new data table obtained by the expansion is added to the first set of candidate routing fields to obtain the second set of candidate routing fields, including: After the capacity is expanded, obtaining the remaining capacity of all data tables in the database; Based on the remaining capacity of each data table, the routing field corresponding to the new data table is added to the first set of candidate routing fields, so that the ratio of the number of routing fields in the second set of candidate routing fields is equal to the ratio of the remaining capacity of the data table corresponding to each routing field.

2. The method according to claim 1, It is characterized in that The step of constructing a first set of to-be-selected routing fields based on each routing field includes: The routing fields corresponding to the data tables are added to the first set of candidate routing fields in equal proportion, so that the number of the routing fields in the first set of candidate routing fields is equal.

3. The method according to claim 1, It is characterized in that The identification field further includes a time field, an Internet Protocol address IP field, and a sequence field; Among them, the time field is used to indicate the generation time of the data, the IP field is used to indicate the IP of the service end that generates the data, and the sequence field is used to indicate the sequence number of the data under the same generation time and IP.

4. The method according to claim 1, It is characterized in that The storing of each data in a corresponding data table based on the data segment identity ID includes: Based on the routing field in the shard identity ID of the data, obtain the corresponding sub-library identifier and the sub-table identifier; Based on the sub-library identifier and the sub-table identifier, the data table where the data is to be stored is determined, and the identification field of the data is stored in the data table corresponding to the data as the primary key value of the data.

5. The method according to claim 1, It is characterized in that The method further comprises: After the expansion, when the amount of data stored in the first target data table among all the new data tables in the database is not less than the second preset amount of data, the number of each routing field in the second set of candidate routing fields is adjusted so that the proportion of the number of routing fields corresponding to the first target data table in the obtained third set of candidate routing fields is reduced, and the third set of candidate routing fields is sent to each business terminal, so that when each business terminal generates data, a shard identity ID of the data is generated based on the third set of candidate routing fields; The second preset data amount is smaller than the first preset data amount.

6. The method according to claim 1, It is characterized in that The method further comprises: After the expansion, when the amount of data stored in the second target data table among all the data tables of the database is not less than the third preset amount of data, the routing field corresponding to the second target data table is deleted from the second set of candidate routing fields to obtain a fourth set of candidate routing fields, and the fourth set of candidate routing fields is sent to each business terminal, so that when each business terminal generates data, a shard identity ID of the data is generated based on the fourth set of candidate routing fields; The third preset data volume is greater than the first preset data volume.

7. A device for expanding the capacity of a database by sub-library and sub-table, It is characterized in that include: A first candidate routing field set acquisition module is used to acquire routing fields composed of sub-library identifiers and sub-table identifiers of each data table currently in use in the database, and to construct a first candidate routing field set based on each routing field, and to send the first candidate routing field set to each service end; A data storage module, used for obtaining a shard identity ID of the data when a business end generates data, and storing each data in a corresponding data table based on the shard identity ID of the data, wherein the shard identity ID includes an identification field and a routing field, and the routing field is obtained from the first set of candidate routing fields by the business end corresponding to each data generation; a second candidate routing field set acquisition module, configured to, when the amount of data stored in each data table in the database is not less than a first preset amount, perform sub-library and sub-table expansion on the database, add a routing field obtained by combining a sub-library identifier and a sub-table identifier of a new data table obtained by the expansion, to the first candidate routing field set to obtain a second candidate routing field set, and send the second candidate routing field set to each service end, so that when each service end generates data, a shard identity ID of the data is generated based on the second candidate routing field set; The second candidate routing field set acquisition module is specifically used to obtain the remaining capacity of all data tables in the database after the expansion; based on the remaining capacity of each data table, add the routing field corresponding to the new data table to the first candidate routing field set, so that the ratio of the number of each routing field in the second candidate routing field set is equal to the ratio of the remaining capacity of the data table corresponding to each routing field.

8. An electronic device, It is characterized in that including memory and processor; The memory stores a computer program; The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images