Relational database peak clipping method and system in cloud computing

Abstract

The invention provides a relational database peak clipping method and system in cloud computing, and the method comprises the steps: obtaining the data volume of data received from a master node by slave nodes in a relational database cluster, activating a load balancing module when the data volume is greater than or equal to a threshold value, selecting one node from the slave nodes as a master node and a slave node, and after the master node and the slave node are successfully selected, starting a load balancing module; suspending data synchronization of the main and auxiliary nodes and other nodes; when the relational database cluster receives write operation, the load balancing module determines a target node to which the write operation is written according to the content of the write operation, if the target node is a main node, the write operation is written into the main node, if the target node is a main auxiliary node, the content is marked in the load balancing module, then the written content is loaded into Memcache, and the content is written into the Memcache; and setting a Memcache elimination mode of the content as a notification mode, and writing the content into the main and auxiliary nodes. According to the method, the purpose of peak clipping can be effectively achieved for high-concurrency access, collapse of the relational database is prevented, and the reliability of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of databases, and in particular to a peak shaving method and system for relational databases in cloud computing. Background Art

[0002] Relational database is a widely used database at present. Many enterprises including e-commerce are using relational database. The performance of relational database is directly related to the performance of the entire website, especially when there is a large concurrency, such as flash sales and rush purchases. There are many ways to improve the speed of relational databases. The most common one is to add cache servers. However, for frequent writes, cache servers need to be constantly updated, which will increase the burden on cache servers. In addition, read-write separation of relational databases is also a common technology used to alleviate the pressure of reading and writing relational databases. The so-called read-write separation means that the read operation is read from the slave node of the relational database cluster, and the write operation is written to the master node. When the write operation occurs, the master node sends the data to the slave node to ensure the data consistency of the relational database cluster. Many enterprises deploy applications on the cloud. If a master node of a relational database is set, when there is high concurrency writing, the database of the master node will also be under great pressure, which will become a bottleneck for user access. How to enable the write operation capability of the relational database on the master node has become the key to improving website performance. Summary of the invention

[0003] Aiming at the problem that a relational database cluster in a cloud computing machine is unable to withstand sudden high concurrent write requests, the present invention provides a peak shaving method for a relational database in cloud computing, the method comprising the following steps:

[0004] Obtaining the amount of data received by the slave node from the master node in the relational database cluster, activating the load balancing module when the amount of data is greater than or equal to a threshold, the load balancing module selecting a node from the slave node as the master and slave node, and suspending data synchronization between the master and slave nodes and other nodes when the master and slave nodes are successfully selected;

[0005] When the relational database cluster receives a write operation, the load balancing module determines the destination node to which the write operation is written based on the content of the write operation. If the destination node is the primary node, the write operation is written to the primary node. If the destination node is the primary and secondary nodes, the content is marked in the load balancing module, and then the written content is loaded into Memcache, the content is set to be eliminated in Memcache in a notification mode, and the content is written to the primary and secondary nodes.

[0006] Preferably, the method also includes: when the amount of data decreases to less than the threshold, the load balancing module sends a notification to the Memcache, and the Memcache allows the Memcache to eliminate the content according to the notification; and writes the data of the primary and secondary nodes to the primary node according to the content marked in the load activation module. After the writing is completed, the load balancing module becomes inactivated, the primary and secondary nodes become slave nodes again, and the data synchronization between the primary and slave nodes is started.

[0007] Preferably, the notification method is specifically: the Memcache allows the content to be eliminated in the Memcache only after receiving the notification information from the load balancing module.

[0008] Preferably, the load balancing module selects a node from the slave nodes as the primary and secondary nodes, specifically:

[0009] The load balancing module records the synchronization time taken by the master node to synchronize data with each slave node each time, calculates the average synchronization time taken by each slave node within a preset time; obtains the number of read operations processed by each slave node within a preset time; and determines a master and slave node from all slave nodes based on the average value and the number of times.

[0010] Preferably, a master and slave node are determined from all slave nodes based on the average value and the number of times, specifically: the maximum value of the average values ​​of all slave nodes is selected, the ratio of the average value of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the number of times is used as the master and slave node; or, the maximum value of the number of times of all slave nodes is selected, the ratio of the number of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the average value is used as the master and slave node.

[0011] In addition, the present invention also provides a relational database peak shaving system in cloud computing, the system includes a load balancing module, and the system also includes the following modules:

[0012] The judgment and selection module is used to obtain the amount of data received by the slave node from the master node in the relational database cluster. When the amount of data is greater than or equal to a threshold, the load balancing module is activated. The load balancing module selects a node from the slave node as the master and slave node. When the master and slave nodes are successfully selected, the data synchronization between the master and slave nodes and other nodes is suspended.

[0013] The peak shaving module is used for when the relational database cluster receives a write operation, the load balancing module determines the destination node to which the write operation is written according to the content of the write operation, if the target node is the primary node, then the write operation is written to the primary node, if the target node is the primary and secondary nodes, then the content is marked in the load balancing module, and then the written content is loaded into Memcache, the content is set to be eliminated in the Memcache notification mode, and the content is written to the primary and secondary nodes. Preferably, the system also includes a data synchronization module, the data synchronization module is used for when the amount of data is reduced to less than the threshold, the load balancing module sends a notification to the Memcache, the Memcache allows the Memcache to eliminate the content according to the notification; and writes the data of the primary and secondary nodes to the primary node according to the content marked in the load activation module, after the writing is completed, the load balancing module becomes inactive, the primary and secondary nodes become slave nodes again, and the data synchronization between the primary and slave nodes is started.

[0014] Preferably, the notification method is specifically: the Memcache allows the content to be eliminated in the Memcache only after receiving the notification information from the load balancing module.

[0015] Preferably, the load balancing module selects a node from the slave nodes as the primary and secondary nodes, specifically:

[0016] The load balancing module records the synchronization time taken by the master node to synchronize data with each slave node each time, calculates the average synchronization time taken by each slave node within a preset time; obtains the number of read operations processed by each slave node within a preset time; and determines a master and slave node from all slave nodes based on the average value and the number of times.

[0017] Preferably, a master and slave node are determined from all slave nodes based on the average value and the number of times, specifically: the maximum value of the average values ​​of all slave nodes is selected, the ratio of the average value of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the number of times is used as the master and slave node; or, the maximum value of the number of times of all slave nodes is selected, the ratio of the number of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the average value is used as the master and slave node.

[0018] Finally, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described above is executed.

[0019] Aiming at the problem that high concurrency causes relational database to become a bottleneck for access to the Web and the like in cloud computing, the existing distributed relational database cluster, a single master server as a database write node is difficult to bear a large amount of data written in a short period of time. The present invention starts a new write node when the amount of data received by the slave node from the master node is greater than or equal to a threshold value, determines the destination node to which the write operation is written according to the content of the write operation, if the target node is the master node, writes it to the master node, if the target node is the master and slave node, marks the content in the load balancing module, then loads the written content into Memcache, sets the content in the Memcache elimination mode to the notification mode, and writes the content to the master and slave nodes, when the amount of data is reduced to less than the threshold value, the load balancing module sends a notification to the Memcache, and the Memcache allows the Memcache to eliminate the content according to the notification; and writes the data of the master and slave nodes to the master node according to the content marked in the load activation module, after the writing is completed, the load balancing module becomes inactive, the master and slave nodes become slave nodes again, and the data synchronization of the master node and the slave node is started. Memcache and primary and secondary nodes are used to buffer some requests for writing to the distributed database, reducing the pressure on the primary server. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is a flow chart of Embodiment 1 of the present invention;

[0022] Figure 2 This is a read-write separation structure diagram in the prior art;

[0023] Figure 3 This is a flow chart of another specific embodiment of embodiment 1 of the present invention. DETAILED DESCRIPTION

[0024] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Example 1

[0027] The present invention provides a peak shaving method for relational database in cloud computing, such as Figure 1 As shown, the method comprises the following steps:

[0028] S1, obtaining the amount of data received by the slave node from the master node in the relational database cluster, and activating the load balancing module when the amount of data is greater than or equal to the threshold. The load balancing module selects a node from the slave node as the master and slave node, and when the master and slave node are successfully selected, suspending data synchronization between the master and slave nodes and other nodes;

[0029] A database cluster consists of multiple nodes running a database. Write operations are written to the master node, and read operations are read from the slave nodes. Figure 2 As shown, this is the read-write separation technology of the database. The master node and the slave node keep data consistent through master-slave replication, that is, the master node synchronizes data to the slave node. In the present invention, the master database runs in the master node, and the slave node runs a slave database. If there is no special explanation, the master node can also be called the master database, and the slave node can also be called the slave database, and the two have the same meaning.

[0030] When there are a large number of write operations, the load of the master node will increase. If the write continues to increase, the master node will become the bottleneck of the entire system, and the amount of data for master-slave replication between the master node and the slave node will also increase. The present invention determines whether peak shaving is required based on the amount of data received by the slave node from the master node. If it is greater than or equal to the threshold, the load balancing module is activated. The load balancing module can be located in a separate server or in the master node. The load balancing module selects a node from multiple slave nodes as the master and slave nodes. Since the master and slave nodes also receive the content of the write operation, in order to prevent data conflicts between the master and slave nodes and the master node, the data synchronization between the master and slave nodes and other nodes is suspended.

[0031] S2, when the relational database cluster receives a write operation, the load balancing module determines the destination node to which the write operation is written according to the content of the write operation. If the target node is the primary node, it is written to the primary node. If the target node is the primary and secondary nodes, the content is marked in the load balancing module, and then the written content is loaded into Memcache, the content is set to be eliminated in Memcache in notification mode, and the content is written to the primary and secondary nodes.

[0032] After the load balancing module is activated, the load balancing module will schedule a portion of the write operations to the primary and secondary nodes. Specifically, the destination node to which the write operation is to be written is determined based on the content of the write operation. In a specific embodiment, the destination node to which the write operation is to be written is determined based on the content of the write operation. Specifically, the load balancing module obtains the frequency of the data table corresponding to the read operation initiated by the application server, and divides the data table of the primary node into two categories based on the frequency. If the write operation is to be written to the first category of tables, the write operation is scheduled to the primary node, that is, the target node is the primary node; if the write operation is to be written to the second category of tables, the write operation is scheduled to the primary and secondary nodes, that is, the target node is the primary and secondary nodes. In a specific embodiment of the classification of a table, the table whose frequency is greater than the second threshold is classified as the second category of tables, otherwise the table is classified as the first category of tables.

[0033] If the target node for writing is the master node, the application server reads data from other slave nodes because the master node will replicate data with other slave nodes. If the target node for writing is the primary and secondary nodes, the primary and secondary nodes will not synchronize data with other nodes, but the data written to the primary and secondary nodes will be loaded into Memcache. Memcache is a cache database that acts as a cache between the application server and the database server. The application server will first read data from Memcache. If it cannot read data, it will read from the slave nodes of the distributed database cluster. In order to prevent Memcache from replacing the content written to the primary and secondary nodes, the elimination method of the content in Memcache is set to notification mode, that is, only when Memcache receives the notification, it is possible to replace the content or data corresponding to the notification according to the LRU algorithm.

[0034] After the load balancing module is activated, the master and slave nodes are equivalent to a separate node, which will not communicate with the master node. As for whether the data will be synchronized to other slave nodes, the present invention does not limit this. In a more specific embodiment, the write operation will be synchronized to other slave nodes; in another more specific embodiment, the write operation will not be synchronized to other slave nodes. Only when the amount of data is reduced to less than the threshold, the data will be synchronized to the master node, and then synchronized to other slave nodes by the master node. Since the master and slave nodes will not synchronize data to the master node, data inconsistency will occur, such as Figure 3 As shown, the method also includes: S3, when the amount of data is reduced to less than the threshold, the load balancing module sends a notification to the Memcache, and the Memcache allows the Memcache to eliminate the content according to the notification; and writes the data of the primary and secondary nodes to the primary node according to the content marked in the load activation module. After the writing is completed, the load balancing module becomes inactivated, the primary and secondary nodes become slave nodes again, and the data synchronization between the primary and slave nodes is started.

[0035] There are multiple slave nodes or slave databases in the distributed database cluster. Some slave nodes are performing a large number of read operations. If this slave node is used as the primary and secondary nodes, it will inevitably affect the ongoing read operations. The load balancing module selects a node from the slave nodes as the primary and secondary nodes, specifically:

[0036] The load balancing module records the synchronization time taken by the master node to synchronize data with each slave node each time, calculates the average synchronization time taken by each slave node within a preset time; obtains the number of read operations processed by each slave node within a preset time; and determines a master and slave node from all slave nodes based on the average value and the number of times.

[0037] When the load balancing module is not activated, the master node will continuously update data to the slave nodes. Due to the performance of different slave nodes and the influence of the network, the update time is different each time the same data is updated. The load balancing module will record the time taken by the master node to synchronize data with each slave node. For example, there are three slave nodes, namely Node A, Node B, and Node C. Within the preset time such as 10 minutes, the load balancing module records two data synchronization processes: the corresponding synchronization time of Node A is: 3s and 5s, and the corresponding read operation is 1000 times. The average synchronization time of Node A within the preset time is: 4s;

[0038] The corresponding synchronization time of Node B is: 5s, 6s, the corresponding read operation is 800 times, and the average synchronization time of Node B within the preset time is: 5.5s;

[0039] The corresponding synchronization time of Node C is 6s and 10s, and the corresponding read operation is 1200 times. The average synchronization time of Node C within the preset time is: 8s.

[0040] In a specific embodiment, determining a primary and secondary node from all the slave nodes according to the average value and the number of times is specifically as follows:

[0041] Select the maximum value of the average value of all slave nodes, take the ratio of the average value of each slave node to the maximum value as the weight, and take the node corresponding to the minimum value of the product of the weight and the number of times as the primary and secondary nodes. Using the above data, the maximum value of the average value is calculated to be 8s, the weight corresponding to Node A is 0.5, the weight corresponding to Node B is 0.6875, and the weight corresponding to Node C is 1. The product of the weight and the number of times is 500 for Node A, 550 for Node B, and 1200 for Node C. The node corresponding to the minimum value of the product, namely Node A, is selected as the primary and secondary nodes. In another embodiment, determining a primary and secondary node from all slave nodes according to the average value and the number of times is specifically as follows:

[0042] The maximum value of the number of all slave nodes is selected, the ratio of the number of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the average value is used as the master and slave nodes.

[0043] Using the above data, the maximum value of the number is 1200, the weight corresponding to Node A is 0.833, the weight corresponding to Node B is 0.667, and the weight corresponding to Node C is 1. The product of the weight and the average value is 3.332 for Node A, 3.6685 for Node B, and 8 for Node C. The node corresponding to the minimum value of the product, Node A, is selected as the primary and secondary nodes.

[0044] Example 2

[0045] The present invention also provides a relational database peak shaving system in cloud computing, the system comprising a load balancing module, and the system further comprising the following modules:

[0046] The judgment and selection module is used to obtain the amount of data received by the slave node from the master node in the relational database cluster. When the amount of data is greater than or equal to a threshold, the load balancing module is activated. The load balancing module selects a node from the slave node as the master and slave node. When the master and slave nodes are successfully selected, the data synchronization between the master and slave nodes and other nodes is suspended.

[0047] The peak shaving module is used for when the relational database cluster receives a write operation, the load balancing module determines the destination node to which the write operation is written according to the content of the write operation, if the target node is the primary node, then the write operation is written to the primary node, if the target node is the primary and secondary nodes, then the content is marked in the load balancing module, and then the written content is loaded into Memcache, the content is set to be eliminated in the Memcache notification mode, and the content is written to the primary and secondary nodes. Preferably, the system also includes a data synchronization module, the data synchronization module is used for when the amount of data is reduced to less than the threshold, the load balancing module sends a notification to the Memcache, the Memcache allows the Memcache to eliminate the content according to the notification; and writes the data of the primary and secondary nodes to the primary node according to the content marked in the load activation module, after the writing is completed, the load balancing module becomes inactive, the primary and secondary nodes become slave nodes again, and the data synchronization between the primary and slave nodes is started.

[0048] Preferably, the notification method is specifically: the Memcache allows the content to be eliminated in the Memcache only after receiving the notification information from the load balancing module.

[0049] Preferably, the load balancing module selects a node from the slave nodes as the primary and secondary nodes, specifically:

[0050] The load balancing module records the synchronization time taken by the master node to synchronize data with each slave node each time, calculates the average synchronization time taken by each slave node within a preset time; obtains the number of read operations processed by each slave node within a preset time; and determines a master and slave node from all slave nodes based on the average value and the number of times.

[0051] Preferably, a master and slave node are determined from all slave nodes based on the average value and the number of times, specifically: the maximum value of the average values ​​of all slave nodes is selected, the ratio of the average value of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the number of times is used as the master and slave node; or, the maximum value of the number of times of all slave nodes is selected, the ratio of the number of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the average value is used as the master and slave node.

[0052] Example 3

[0053] The present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in Example 1 is executed.

[0054] Example 4

[0055] The present invention also provides a device, comprising a computer-readable storage medium and a processor, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method described in Example 1 is executed.

[0056] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalent; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A peak shaving method for relational database in cloud computing, characterized in that: The method comprises the following steps: Obtaining the amount of data received by the slave node from the master node in the relational database cluster, activating the load balancing module when the amount of data is greater than or equal to a threshold, the load balancing module selecting a node from the slave node as the master and slave node, and suspending data synchronization between the master and slave nodes and other nodes when the master and slave nodes are successfully selected; When the relational database cluster receives a write operation, the load balancing module determines the destination node to which the write operation is written based on the content of the write operation. If the destination node is the primary node, the write operation is written to the primary node. If the destination node is the primary and secondary nodes, the content is marked in the load balancing module, and then the written content is loaded into Memcache, the content is set to be eliminated in Memcache in a notification mode, and the content is written to the primary and secondary nodes.

2. The method according to claim 1, characterized in that The method also includes: when the amount of data decreases to less than the threshold, the load balancing module sends a notification to the Memcache, and the Memcache allows the Memcache to eliminate the content according to the notification; and writes the data of the primary and secondary nodes to the primary node according to the content marked in the load activation module. After the writing is completed, the load balancing module becomes inactive, the primary and secondary nodes become slave nodes again, and the data synchronization between the primary and slave nodes is started.

3. The method according to claim 1, characterized in that The notification method is specifically: the Memcache is allowed to eliminate the content only after receiving the notification information from the load balancing module.

4. The method according to claim 1, characterized in that The load balancing module selects a node from the slave nodes as the primary and secondary nodes, specifically: The load balancing module records the synchronization time taken by the master node to synchronize data with each slave node each time, calculates the average synchronization time taken by each slave node within a preset time; obtains the number of read operations processed by each slave node within a preset time; A master and slave node is determined from all slave nodes according to the average value and the number of times.

5. The method according to claim 4, characterized in that The step of determining a primary and secondary node from all the secondary nodes according to the average value and the number of times is specifically as follows: The maximum value of the average values ​​of all slave nodes is selected, the ratio of the average value of each slave node to the maximum value is used as a weight, and the node corresponding to the minimum value of the product of the weight and the number of times is used as the primary and secondary nodes; or, The maximum value of the number of all slave nodes is selected, the ratio of the number of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the average value is used as the master and slave nodes.

6. A relational database peak shaving system in cloud computing, characterized in that: The system includes a load balancing module, and the system also includes the following modules: The judgment and selection module is used to obtain the amount of data received by the slave node from the master node in the relational database cluster. When the amount of data is greater than or equal to a threshold, the load balancing module is activated. The load balancing module selects a node from the slave node as the master and slave node. When the master and slave nodes are successfully selected, the data synchronization between the master and slave nodes and other nodes is suspended. The peak shaving module is used for when the relational database cluster receives a write operation, the load balancing module determines the destination node to which the write operation is written according to the content of the write operation, if the target node is the primary node, then the write operation is written into the primary node, if the target node is the primary and secondary nodes, then the content is marked in the load balancing module, and then the written content is loaded into Memcache, the content is set to be eliminated in Memcache in a notification mode, and the content is written to the primary and secondary nodes.

7. The system according to claim 6, characterized in that The system also includes a data synchronization module, which is used to, when the amount of data decreases to less than the threshold, cause the load balancing module to send a notification to the Memcache, and the Memcache allows the Memcache to eliminate the content according to the notification; and write the data of the primary and secondary nodes to the primary node according to the content marked in the load activation module. After the writing is completed, the load balancing module becomes inactive, the primary and secondary nodes become slave nodes again, and the data synchronization between the primary and slave nodes is started.

8. The system according to claim 6, characterized in that The load balancing module selects a node from the slave nodes as the primary and secondary nodes, specifically: The load balancing module records the synchronization time taken by the master node to synchronize data with each slave node each time, calculates the average synchronization time taken by each slave node within a preset time; obtains the number of read operations processed by each slave node within a preset time; A master and slave node is determined from all slave nodes according to the average value and the number of times.

9. The system according to claim 8, characterized in that The step of determining a primary and secondary node from all the secondary nodes according to the average value and the number of times is specifically as follows: The maximum value of the average values ​​of all slave nodes is selected, the ratio of the average value of each slave node to the maximum value is used as a weight, and the node corresponding to the minimum value of the product of the weight and the number of times is used as the primary and secondary nodes; or, The maximum value of the number of all slave nodes is selected, the ratio of the number of each slave node to the maximum value is used as the weight, and the node corresponding to the minimum value of the product of the weight and the average value is used as the master and slave nodes.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is performed.

Images