Database capacity expansion method and device, electronic equipment and storage medium

Abstract

The invention discloses a database capacity expansion method and device, electronic equipment and a storage medium, relates to a distributed technology, is applied to middleware of a distributed database, and comprises the following steps: copying according to a sample node to obtain a copied node; dividing the affiliation of the logic fragments associated with the sample nodes according to the copy nodes and the sample nodes to obtain a logic mapping relation; adjusting a routing mapping rule according to the logic mapping relation; determining an accessed target logic fragment according to the obtained structured query statement; determining routing information of the target logic fragment according to the adjusted routing mapping rule; and performing data processing of the database instance corresponding to the target logic fragment according to the routing information. Due to the fact that capacity expansion is carried out in the mode of copying the sample nodes, the computing power of each node is matched with the storage space, load balancing of the distributed database is achieved, each node has reasonable throughput capacity, and the operation efficiency of distributed data is improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, and in particular to a database expansion method, device, electronic equipment and storage medium. Background Art

[0002] As the volume of system business increases, the pressure on database storage will gradually increase. In a distributed database, when the storage of a single storage node reaches a performance bottleneck, the business efficiency and timeliness of the storage will be reduced. For the above problem, the current approach is to increase the storage capacity of each storage node. For example, increase the storage space of the storage node.

[0003] However, a single storage node has an upper limit on storage space and cannot be increased indefinitely. In addition, the computing power of a single node is limited. When too much data is stored, it cannot respond quickly to data reading and writing. Therefore, how to efficiently expand the capacity of a distributed database has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a database expansion method, device, electronic device and storage medium to solve the problem that it is currently impossible to achieve shared storage of different types of databases without performing data migration.

[0005] According to one aspect of the present invention, a database expansion method is provided, which is applied to middleware of a distributed database, comprising:

[0006] Copy the sample node to obtain a copy node;

[0007] The ownership of the logical shards associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain a logical mapping relationship;

[0008] Adjusting the routing mapping rules according to the logical mapping relationship;

[0009] Determine the target logical shard to be accessed based on the obtained structured query statement;

[0010] Determine the routing information of the target logical slice according to the adjusted routing mapping rule;

[0011] Data processing of the database instance corresponding to the target logical shard is performed according to the routing information.

[0012] According to another aspect of the present invention, there is provided a database expansion device, which is applied to middleware of a distributed database, and comprises:

[0013] A replication module, used to replicate the sample node to obtain a replication node;

[0014] A logical mapping module is used to divide the ownership of the logical slices associated with the sample node according to the replication node and the sample node to obtain a logical mapping relationship;

[0015] A routing adjustment module, used for adjusting routing mapping rules according to the logical mapping relationship;

[0016] The parsing module is used to determine the target logical shard to be accessed based on the obtained structured query statement;

[0017] A routing determination module, used to determine the routing information of the target logical slice according to the adjusted routing mapping rule;

[0018] A data processing module is used to perform data processing on the database instance corresponding to the target logical shard according to the routing information.

[0019] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the database expansion method described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the database expansion method described in any embodiment of the present invention when executed.

[0024] The technical solution of the embodiment of the present invention is to replicate the sample node to obtain the replicated node; to divide the ownership of the logical shard associated with the sample node according to the replicated node and the sample node to obtain the logical mapping relationship; to adjust the routing mapping rule according to the logical mapping relationship; to determine the target logical shard to be accessed according to the obtained structured query statement; to determine the routing information of the target logical shard according to the adjusted routing mapping rule; and to process the data of the database instance corresponding to the target logical shard according to the routing information. Compared with the current expansion of the storage capacity of a single storage node, the technical solution provided by the embodiment of the present invention can achieve the capacity expansion of a horizontal distributed database by replicating the sample node to obtain the replicated node. After the capacity expansion is performed, the ownership of the logical shard is divided, and after obtaining the new logical mapping, the mapping relationship in each node is reduced. When the mapping relationship on the node is reduced, the available storage space of the database instance corresponding to each logical shard increases, thereby achieving the expansion of the storage space of the database instance. When a structured query statement is received, the routing information is determined according to the logical mapping relationship, and the data of the database instance is processed according to the routing information, which can achieve rapid expansion of the database and efficient use of the expanded database, and achieve efficient expansion of the distributed database and rapid deployment and use of the expanded database, thereby improving the resource utilization of distributed data. Since the expansion is carried out by replicating sample nodes, the computing power of each node matches the storage space, achieving load balancing of the distributed database, so that each node has a reasonable throughput, and improving the operating efficiency of distributed data.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a schematic diagram of a system architecture of a distributed database provided by an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of database nodes of a distributed database provided by an embodiment of the present invention;

[0029] Figure 3 It is a flowchart of a method for expanding database capacity provided by an embodiment of the present invention;

[0030] Figure 4 It is a schematic diagram of database node content for database expansion provided by an embodiment of the present invention;

[0031] Figure 5 It is a flowchart of another database expansion method provided by an embodiment of the present invention;

[0032] Figure 6 It is a structural schematic diagram of a database expansion device provided by an embodiment of the present invention;

[0033] Figure 7 It is a structural schematic diagram of an electronic device for implementing the database expansion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] The information collected by the embodiments of the present invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation portals for users to choose to authorize or refuse. Provide users with corresponding operation portals for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered to avoid relevant legal risks and public opinion risks.

[0037] As the volume of system business increases, the pressure on database storage will gradually increase. In a distributed database, when the storage of a single storage node reaches a performance bottleneck, the business efficiency and timeliness of the storage will be reduced. For the above problem, the current approach is to increase the storage capacity of each storage node. For example, by expanding the capacity of the storage node's disk, etc., the storage space of the storage node can be increased.

[0038] However, a single storage node has an upper limit on storage space, such as the upper limit of disk specifications and the number of expandable disk slots, so it cannot be increased indefinitely. In addition, the computing power of a single node is limited. When too much data is stored, it cannot respond quickly to data reading and writing. As the storage space of a single node increases, the time for a full disk scan increases. Since the processor computing power is fixed, as the storage space of a single node continues to increase, its data search ability decreases.

[0039] In the scenario of using existing distributed databases, the data sharding algorithm can be used to split massive data according to the specified sharding key to achieve the purpose of storage and traffic load balancing. When using the consistent hashing algorithm, the original data path can be maintained by adding or deleting one of the nodes. However, the consistent hashing algorithm has certain limitations, such as high complexity, increased consistency guarantee cost, and high maintenance complexity. In the inconsistent hashing algorithm, since the sharding structure is fixed during system design, if a node is added to the distributed cluster, the full amount of data in the cluster needs to be redistributed, which becomes very complicated under the premise of today's business continuity requirements.

[0040] Therefore, how to efficiently expand the capacity of distributed databases has become an urgent problem to be solved.

[0041] Figure 1 The system architecture diagram of the distributed database provided for the embodiment of the present invention includes a client, a middleware and a plurality of database nodes. The client sends a structured query statement (Structured Query Language, SQL) to the middleware, and the query statement includes the accessed data table and the operation information of the data in the data table. The middleware puts the structured query statement into the database node for execution according to the structured query statement and the established routing information. In this solution, the middleware expands the capacity of the database nodes in the distributed database, and the expansion includes node replication, generation of new mapping relationships and generation of new routing information.

[0042] like Figure 2As shown, each database node is a physical node, and the database node includes multiple data shards, and the data shards correspond to the logical shards one by one. The data shards are used to store the data of the logical shards, and the logical shards can be the identifiers of the data shards, which are used to represent the identifiers of the data shards when using structured query statements to perform data operations. For example, Figure 2 As shown, a database node includes four data shards (dataHost), each data shard stores the data of a database instance, each data shard is represented by a logical shard (dataNode), and the four data shards correspond to four logical shards respectively.

[0043] Figure 3 This is a flow chart of a database expansion method provided by an embodiment of the present invention. This embodiment is applicable to the case of expanding a distributed database and providing database services based on the expanded database, especially for the case of expanding a distributed database using a non-consistent hashing algorithm. The method can be executed by a database expansion device, which can be implemented in the form of hardware and / or software, and can be configured in electronic devices such as personal computers and servers. Figure 3 As shown, the method is applied to the middleware of the distributed database, including:

[0044] S101. Duplicate a sample node to obtain a duplicate node.

[0045] The distributed database includes multiple database nodes, and the sample node is any database node. When the distributed data is expanded horizontally, each database node in the distributed database can be replicated. The replication method can be full replication, which copies all the big data in the sample node to the replication node, and the data content in the replication node is the same as that of the sample node. The sample node can be replicated to obtain one or more replication nodes.

[0046] Optionally, the sample node is copied to obtain a copy node, including:

[0047] The number of copies N is determined according to the demand information; and the sample node is copied according to the number of copies N to obtain N copy nodes.

[0048] The value of N should satisfy the following conditions: N+1 is an exponential power of 2 and N+1≤the number of logical shards contained in the sample node.

[0049] Analyze the load of the distributed database. The number of replications N can be 1, 3, 7, etc. Assume that the number of replications is N, N+1=2 m , m is an integer greater than 0.

[0050] The demand information can be determined based on the load status of the distributed database. For example, if a database node has a performance bottleneck and needs to be expanded, the demand information is determined based on the throughput and performance indicators of the database node. The capacity to be expanded can also be determined by the database node, such as expanding to 1 times, 2 times, or n times the current capacity. The demand information is used to indicate the number of copies that need to be replicated. After replication according to the number of copies, the sum of the number of replication nodes and sample nodes is an exponential power of 2.

[0051] The total number of replication nodes and sample nodes is less than or equal to the number of logical shards contained in the sample node. When the total number of replication nodes and sample nodes is equal to the number of logical shards contained in the sample node, each database node is used to store a data shard corresponding to a logical shard.

[0052] Optionally, the number of sample nodes is an exponential power of 2; the number of logical shards contained in the sample nodes is an exponential power of 2.

[0053] The above implementation method can reasonably expand capacity according to demand, avoiding over-expansion or under-expansion. In addition, controlling the number of replications according to the exponential power of 2 can make the number of nodes obtained by expansion easier to manage.

[0054] S102: Divide the ownership of the logical slices associated with the sample nodes according to the replication nodes and the sample nodes to obtain a logical mapping relationship.

[0055] After replication, the replication node has data shards corresponding to all logical shards in the sample node. By configuring the ownership of the roadbed shards, you can determine which data shards in the replication node and sample node are valid and which are invalid. If a logical shard belongs to a node, it will not belong to any other node. By dividing the ownership of the logical shards, you can get the logical shards associated with each node (sample node or replication node), that is, the node to which the logical shard belongs.

[0056] Optionally, the ownership of the logical shards associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain a logical mapping relationship, which can be implemented in the following manner:

[0057] Determine the number of logical equal divisions according to the sum of the number of replication nodes and the number of sample nodes; calculate the number of equal divisions according to the number of logical equal divisions and the sum of the number of logical shards; and evenly distribute the logical shards to the replication nodes and the sample nodes according to the number of equal divisions.

[0058] When performing logical sharding, you can use an even distribution method. The sum of the number of replication nodes and sample nodes is used as the number of logical equal divisions. The sum of the number of logical shards is divided by the number of logical equal divisions to obtain the equal division number. Allocate logical shards to each node according to the equal division number.

[0059] Furthermore, after the ownership of the logical slices associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain the logical mapping relationship, the method further includes:

[0060] According to the logical mapping relationship, the logical shards to which the logical mapping relationship is not assigned are deleted in the replication node and the sample node.

[0061] Since the replication node has all the data in the sample node, in order to prevent the data shards of non-associated logical shards from occupying storage space, after determining the logical mapping relationship, the logical shards to which the logical mapping relationship is not assigned and their corresponding data shards are deleted.

[0062] For example, Figure 4 The database expansion diagram provided in the embodiment of the present invention assumes that there are 4 logical shards in the sample node, corresponding to 4 data shards. The sample node is set0, and the replication node is set16. By evenly distributing, logical shard 0 and logical shard 1 are allocated to the sample node, and logical shard 2 and logical shard 3 are allocated to the replication node. Then, the data shards corresponding to logical shard 2 and logical shard 3 in the sample node are deleted, and the data shards corresponding to logical shard 0 and logical shard 1 in the replication node are deleted.

[0063] Optionally, the ownership of the logical shards associated with the sample node is divided according to the replication node and the sample node to obtain a logical mapping relationship, further comprising:

[0064] In the logical mapping relationship, the copy node is used as a child node of the sample node;

[0065] The sample node is associated with a plurality of standby sample nodes, the standby sample nodes are child nodes of the sample node, and the data in the standby sample nodes is the same as the data in the sample node;

[0066] The replication node is associated with a plurality of standby replication nodes, and the standby replication nodes are child nodes of the replication node.

[0067] Each database node in a distributed database can have a master-slave solution, that is, each node serves as a master node and has three standby nodes. The standby node communicates with the master node regularly to complete data synchronization. When the master node fails, one of the standby nodes is selected as a new master node. In an embodiment of the present invention, in a logical mapping relationship, a copy node can be used as a child node of a sample node. At this time, the child nodes of the sample node include a copy node and multiple standby nodes. At the same time, the copy node has multiple standby nodes as child nodes.

[0068] The above implementation method can configure a backup node for each node, thereby improving the reliability of the database through the master-slave mode.

[0069] S103: Adjust the routing mapping rules according to the logical mapping relationship.

[0070] S104: Determine the target logical shard to be accessed according to the obtained structured query statement.

[0071] The database horizontal expansion provided by the embodiment of the present invention can realize online real-time expansion. Since there is no need to reconfigure the distribution mode of the distributed nodes, the database can be expanded while ensuring the operation of the distributed database. After updating the logical mapping relationship and routing rules, the database data processing service can be quickly provided to the client.

[0072] S105. Determine routing information of the target logical slice according to the adjusted routing mapping rule.

[0073] S106. Perform data processing on the database instance corresponding to the target logical shard according to the routing information.

[0074] Optionally, performing data processing of the database instance corresponding to the target logical shard according to the routing information may be implemented in the following manner:

[0075] Obtain idle connections established in the connection pool according to the replication node and the sample node; and perform data processing of the database instance corresponding to the target logical shard according to the routing information through the idle connections.

[0076] The connection pool has established communication connections, and the physical sharding corresponding to the logical sharding on the target database node can be performed through the idle connection. The above implementation method can improve the database node access speed and improve the data processing efficiency.

[0077] The database expansion method of the embodiment of the present invention is to replicate according to the sample node to obtain the replication node; to divide the ownership of the logical shard associated with the sample node according to the replication node and the sample node to obtain the logical mapping relationship; to adjust the routing mapping rule according to the logical mapping relationship; to determine the target logical shard to be accessed according to the obtained structured query statement; to determine the routing information of the target logical shard according to the adjusted routing mapping rule; and to perform data processing of the database instance corresponding to the target logical shard according to the routing information. Compared with the current expansion of the storage capacity of a single storage node, the database expansion method provided by the embodiment of the present invention can achieve the capacity expansion of a horizontal distributed database by replicating the sample node to obtain the replication node. After the capacity expansion is performed, the ownership of the logical shard is divided, and after obtaining the new logical mapping, the mapping relationship in each node is reduced. When the mapping relationship on the node is reduced, the available storage space of the database instance corresponding to each logical shard increases, thereby achieving the expansion of the storage space of the database instance. When a structured query statement is received, the routing information is determined according to the logical mapping relationship, and the data of the database instance is processed according to the routing information, which can achieve rapid expansion of the database and efficient use of the expanded database, and achieve efficient expansion of the distributed database and rapid deployment and use of the expanded database, thereby improving the resource utilization of distributed data. Since the expansion is carried out by replicating sample nodes, the computing power of each node matches the storage space, achieving load balancing of the distributed database, so that each node has a reasonable throughput, and improving the operating efficiency of distributed data.

[0078] Figure 5 A schematic diagram of a database expansion method according to an embodiment of the present invention, comprising:

[0079] S201. Determine a replication number N according to demand information; replicate a sample node according to the replication number N to obtain N replication nodes.

[0080] The value of N should satisfy: N+1 is an exponential power of 2 and N+1≤the number of logical shards contained in the sample node. The number of sample nodes is an exponential power of 2; the number of logical shards contained in the sample node is an exponential power of 2.

[0081] S202, determine the number of logical equal divisions according to the total number of replication nodes and sample nodes; calculate the number of equal divisions according to the number of logical equal divisions and the total number of logical shards; and evenly distribute the logical shards to the replication nodes and sample nodes according to the number of equal divisions.

[0082] In the logical mapping relationship, the copy node is used as a child node of the sample node; the sample node is associated with multiple standby sample nodes, the standby sample nodes are child nodes of the sample node, and the data in the standby sample nodes are the same as the data in the sample node; the copy node is associated with multiple standby copy nodes, and the standby copy nodes are child nodes of the copy node.

[0083] S203. According to the logical mapping relationship, delete the logical slices to which the logical mapping relationship is not assigned in the replication node and the sample node.

[0084] S204: Adjust the routing mapping rules according to the logical mapping relationship.

[0085] S205: Determine the target logical shard to be accessed according to the obtained structured query statement.

[0086] S206. Determine routing information of the target logical slice according to the adjusted routing mapping rule.

[0087] S207. Obtain idle connections established in the connection pool according to the replication nodes and the sample nodes; and perform data processing on the database instance corresponding to the target logical shard according to the routing information through the idle connections.

[0088] The database expansion method of the embodiment of the present invention is to replicate according to the sample node to obtain the replication node; to divide the ownership of the logical shard associated with the sample node according to the replication node and the sample node to obtain the logical mapping relationship; to adjust the routing mapping rule according to the logical mapping relationship; to determine the target logical shard to be accessed according to the obtained structured query statement; to determine the routing information of the target logical shard according to the adjusted routing mapping rule; and to perform data processing of the database instance corresponding to the target logical shard according to the routing information. Compared with the current expansion of the storage capacity of a single storage node, the database expansion method provided by the embodiment of the present invention can achieve the capacity expansion of a horizontal distributed database by replicating the sample node to obtain the replication node. After the capacity expansion is performed, the ownership of the logical shard is divided, and after obtaining the new logical mapping, the mapping relationship in each node is reduced. When the mapping relationship on the node is reduced, the available storage space of the database instance corresponding to each logical shard increases, thereby achieving the expansion of the storage space of the database instance. When a structured query statement is received, the routing information is determined according to the logical mapping relationship, and the data of the database instance is processed according to the routing information, which can achieve rapid expansion of the database and efficient use of the database after expansion, and achieve efficient expansion of the distributed database and rapid deployment and use of the database after expansion, thereby improving the resource utilization of distributed data. Since the expansion is carried out by replicating sample nodes, the computing power of each node matches the storage space, and the load balancing of the distributed database is achieved, so that each node has a reasonable throughput and improves the operating efficiency of distributed data. In the distributed database scenario under the non-consistent hashing algorithm, there is no need to fully redistribute the existing data, and the horizontal expansion of data can be completed efficiently.

[0089] Figure 6 1 is a schematic diagram of a structure of a database expansion device provided by an embodiment of the present invention. This embodiment is applicable to the case of expanding a distributed database and providing database services based on the expanded database, especially for the case of expanding a distributed database using a non-consistent hashing algorithm, such as Figure 6 As shown, the device includes: a replication module 31, a logic mapping module 32, a route adjustment module 33, a parsing module 34, a route determination module 35 and a data processing module 36.

[0090] A replication module 31, used for replicating the sample node to obtain a replication node;

[0091] The logic mapping module 32 is used to divide the ownership of the logical slices associated with the sample nodes according to the replication nodes and the sample nodes to obtain a logic mapping relationship;

[0092] A routing adjustment module 33, configured to adjust routing mapping rules according to the logical mapping relationship;

[0093] The parsing module 34 is used to determine the target logical shard to be accessed according to the obtained structured query statement;

[0094] A routing determination module 35, configured to determine routing information of the target logical slice according to the adjusted routing mapping rule;

[0095] The data processing module 36 is used to perform data processing on the database instance corresponding to the target logical shard according to the routing information.

[0096] Based on the above embodiment, optionally, the logic mapping module 32 is used to:

[0097] Determine the number of logical equal divisions according to the sum of the number of replicated nodes and sample nodes;

[0098] Calculate the equal number of shares according to the sum of the number of logical equal shares and the number of logical shards;

[0099] The logical shards are evenly distributed to the replication nodes and the sample nodes according to the equal distribution amount.

[0100] Based on the above embodiment, optionally, the copy module 31 is used to:

[0101] Determine the number of copies N based on the demand information;

[0102] The sample node is replicated according to the replication quantity to obtain N replication nodes;

[0103] The value of N should satisfy the following conditions: N+1 is an exponential power of 2 and N+1≤the number of logical shards contained in the sample node.

[0104] Based on the above embodiment, optionally, the logic mapping module 32 is further used for:

[0105] According to the logical mapping relationship, the logical shards to which the logical mapping relationship is not assigned are deleted in the replication node and the sample node.

[0106] Based on the above embodiment, optionally, the logic mapping module 32 is further used for:

[0107] In the logical mapping relationship, the copy node is used as a child node of the sample node;

[0108] The sample node is associated with a plurality of standby sample nodes, the standby sample nodes are child nodes of the sample node, and the data in the standby sample nodes is the same as the data in the sample node;

[0109] The replication node is associated with a plurality of standby replication nodes, and the standby replication nodes are child nodes of the replication node.

[0110] Based on the above embodiment, optionally, the data processing module 36 is used to:

[0111] Get the idle connections established in the connection pool based on the replication node and sample node;

[0112] Data processing of the database instance corresponding to the target logical shard is performed through the idle connection according to the routing information.

[0113] Based on the above embodiment, optionally, the number of the sample nodes is an exponential power of 2;

[0114] The number of logical shards included in the sample node is an exponential power of 2.

[0115] The database expansion device of the embodiment of the present invention comprises a replication module 31, which is used to replicate according to the sample node to obtain the replication node; a logic mapping module 32, which is used to divide the ownership of the logical shards associated with the sample node according to the replication node and the sample node to obtain the logic mapping relationship; a routing adjustment module 33, which is used to adjust the routing mapping rule according to the logic mapping relationship; a parsing module 34, which is used to determine the target logical shard to be accessed according to the acquired structured query statement; a routing determination module 35, which is used to determine the routing information of the target logical shard according to the adjusted routing mapping rule; and a data processing module 36, which is used to perform data processing of the database instance corresponding to the target logical shard according to the routing information. Compared with the current expansion of the storage capacity of a single storage node, the database expansion device provided by the embodiment of the present invention can achieve the capacity expansion of a horizontal distributed database by replicating the sample node to obtain the replication node. After the capacity expansion is performed, the ownership of the logical shards is divided, and after obtaining the new logical mapping, the mapping relationship in each node is reduced. When the mapping relationship on the node is reduced, the available storage space of the database instance corresponding to each logical shard is increased, so as to achieve the expansion of the storage space of the database instance. When a structured query statement is received, the routing information is determined according to the logical mapping relationship, and the data of the database instance is processed according to the routing information, which can achieve rapid expansion of the database and efficient use of the database after expansion, and achieve efficient expansion of the distributed database and rapid deployment and use of the database after expansion, thereby improving the resource utilization of distributed data. Since the expansion is carried out by replicating sample nodes, the computing power of each node matches the storage space, and the load balancing of the distributed database is achieved, so that each node has a reasonable throughput and improves the operating efficiency of distributed data. In the distributed database scenario under the non-consistent hashing algorithm, there is no need to fully redistribute the existing data, and the horizontal expansion of data can be completed efficiently.

[0116] The database expansion device provided in the embodiment of the present invention can execute the database expansion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0117] Figure 7 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0118] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a database expansion method.

[0121] In some embodiments, the database expansion method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the database expansion method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the database expansion method in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The computer program for implementing the database expansion method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed completely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or completely on a remote machine or server.

[0124] The embodiment of the invention further provides a computer-readable storage medium, the computer-readable storage medium storing computer instructions, the computer instructions being used to cause a processor to execute a database expansion method, applied to a middleware of a distributed database, comprising:

[0125] Copy the sample node to obtain a copy node;

[0126] The ownership of the logical shards associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain a logical mapping relationship;

[0127] Adjusting the routing mapping rules according to the logical mapping relationship;

[0128] Determine the target logical shard to be accessed based on the obtained structured query statement;

[0129] Determine the routing information of the target logical slice according to the adjusted routing mapping rule;

[0130] Data processing of the database instance corresponding to the target logical shard is performed according to the routing information.

[0131] On the basis of the above embodiment, optionally, the ownership of the logical slices associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain a logical mapping relationship, including:

[0132] Determine the number of logical equal divisions according to the sum of the number of replicated nodes and sample nodes;

[0133] Calculate the equal number of shares according to the sum of the number of logical equal shares and the number of logical shards;

[0134] The logical shards are evenly distributed to the replication nodes and the sample nodes according to the equal distribution amount.

[0135] Based on the above embodiment, optionally, duplicating the sample node to obtain the duplicate node includes:

[0136] Determine the number of copies N based on the demand information;

[0137] The sample node is replicated according to the replication quantity N to obtain N replication nodes;

[0138] The value of N should satisfy the following conditions: N+1 is an exponential power of 2 and N+1≤the number of logical shards contained in the sample node.

[0139] Based on the above embodiment, optionally, after the ownership of the logical slices associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain the logical mapping relationship, the method further includes:

[0140] According to the logical mapping relationship, the logical shards to which the logical mapping relationship is not assigned are deleted in the replication node and the sample node.

[0141] On the basis of the above embodiment, optionally, the ownership of the logical slices associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain the logical mapping relationship, further comprising:

[0142] In the logical mapping relationship, the copy node is used as a child node of the sample node;

[0143] The sample node is associated with a plurality of standby sample nodes, the standby sample nodes are child nodes of the sample node, and the data in the standby sample nodes is the same as the data in the sample node;

[0144] The replication node is associated with a plurality of standby replication nodes, and the standby replication nodes are child nodes of the replication node.

[0145] Based on the above embodiment, optionally, performing data processing of the database instance corresponding to the target logical shard according to the routing information includes:

[0146] Get the idle connections established in the connection pool based on the replication node and sample node;

[0147] Data processing of the database instance corresponding to the target logical shard is performed through the idle connection according to the routing information.

[0148] Based on the above embodiment, optionally, the number of the sample nodes is an exponential power of 2;

[0149] The number of logical shards included in the sample node is an exponential power of 2.

[0150] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0153] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0154] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0155] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for expanding database capacity, characterized in that: Middleware for distributed databases, including: Copy the sample node to obtain a copy node; The ownership of the logical shards associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain a logical mapping relationship; Adjusting the routing mapping rules according to the logical mapping relationship; Determine the target logical shard to be accessed based on the structured query statement obtained; Determine the routing information of the target logical slice according to the adjusted routing mapping rule; Data processing of the database instance corresponding to the target logical shard is performed according to the routing information.

2. The method according to claim 1, characterized in that The ownership of the logical shards associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain a logical mapping relationship, including: Determine the number of logical equal divisions according to the sum of the number of replicated nodes and sample nodes; Calculate the equal number of shares according to the sum of the number of logical equal shares and the number of logical shards; The logical shards are evenly distributed to the replication nodes and the sample nodes according to the equal distribution amount.

3. The method according to claim 2, characterized in that Copy the sample node to obtain the copy node, including: Determine the number of copies N based on the demand information; The sample node is replicated according to the replication quantity N to obtain N replication nodes; The value of N should satisfy the following conditions: N+1 is an exponential power of 2 and N+1≤the number of logical shards contained in the sample node.

4. The method according to claim 2, characterized in that: After the ownership of the logical shards associated with the sample nodes is divided according to the replication nodes and the sample nodes to obtain the logical mapping relationship, the following is also included: According to the logical mapping relationship, the logical shards to which the logical mapping relationship is not assigned are deleted in the replication node and the sample node.

5. The method according to claim 1, characterized in that: The logical shards associated with the sample nodes are divided according to the replication nodes and the sample nodes to obtain a logical mapping relationship, which also includes: In the logical mapping relationship, the copy node is used as a child node of the sample node; The sample node is associated with a plurality of standby sample nodes, the standby sample nodes are child nodes of the sample node, and the data in the standby sample nodes is the same as the data in the sample node; The replication node is associated with a plurality of standby replication nodes, and the standby replication nodes are child nodes of the replication node.

6. The method according to claim 1, characterized in that Performing data processing of the database instance corresponding to the target logical shard according to the routing information includes: Get the idle connections established in the connection pool based on the replication node and sample node; Data processing of the database instance corresponding to the target logical shard is performed through the idle connection according to the routing information.

7. The method according to claim 1, characterized in that The number of sample nodes is an exponential power of 2; The number of logical shards included in the sample node is an exponential power of 2.

8. A database expansion device, characterized in that: Middleware for distributed databases, including: A replication module, used to replicate the sample node to obtain a replication node; A logical mapping module is used to divide the ownership of the logical slices associated with the sample node according to the replication node and the sample node to obtain a logical mapping relationship; A routing adjustment module, used for adjusting routing mapping rules according to the logical mapping relationship; The parsing module is used to determine the target logical shard to be accessed based on the obtained structured query statement; A routing determination module, used to determine the routing information of the target logical slice according to the adjusted routing mapping rule; A data processing module is used to perform data processing on the database instance corresponding to the target logical shard according to the routing information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the database expansion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the database expansion method according to any one of claims 1 to 7 when executed.

Images