Method and device for database cluster capacity expansion, electronic equipment, storage medium and program product
By monitoring the storage space utilization and data growth rate of the database cluster, and automatically expanding capacity, the problem of not being able to dynamically adjust storage resources in the existing technology is solved, and the response speed and expansion capabilities of the database cluster are improved.
Patent Information
- Application Number
- CN202411111608.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-08
AI Technical Summary
Existing database capacity expansion methods cannot dynamically adjust storage resources based on actual data growth, rely on manual triggering or timed scripts, and cannot respond to data storage needs in real time.
By obtaining the storage space utilization rate and data storage growth rate of the database cluster, it is automatically judged whether to expand capacity, including expansion of storage nodes and replica nodes or slice expansion, ensuring dynamic adjustment of storage resources.
It realizes dynamic adjustment of storage resources based on actual data growth, improves the response speed and expansion capabilities of the database cluster, and reduces data storage failures caused by lag in capacity.
Smart Images

Figure CN118916346A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data management technology, and in particular to a method, device, electronic device, storage medium and program product for expanding the capacity of a database cluster. Background Art
[0002] With the advent of the big data era, the demand for data storage and processing is growing. When the existing database cannot meet business needs, capacity expansion is required. Existing database capacity expansion methods usually require manual triggering or rely on timed scripts, and cannot dynamically adjust storage resources according to actual data growth. Summary of the invention
[0003] The embodiments of the present application provide a method, an apparatus, an electronic device, a storage medium, and a program product for expanding the capacity of a database cluster, so as to achieve the effect of dynamically adjusting storage resources according to actual data growth.
[0004] In a first aspect, an embodiment of the present application provides a method for expanding a database cluster, including: obtaining a storage space utilization rate and a data storage growth rate of a database cluster; the data storage growth rate is a data growth rate within a set time; the set time is a time period corresponding to a maximum data growth amount within a preset time period; if the data storage growth rate is greater than a set growth rate, and the storage space utilization rate is greater than or equal to a set space utilization rate, the database cluster is expanded.
[0005] Optionally, in the method as described above, the database cluster includes storage nodes and replica nodes corresponding to each storage node; expanding the database cluster includes: if the storage space utilization rate is greater than the set space utilization rate and less than the maximum set utilization rate, then expanding the storage capacity of each storage node and each replica node; or, if the storage space utilization rate is equal to the maximum set utilization rate, then slicing the database cluster.
[0006] Optionally, the method as described above, slices and expands the database cluster, including: obtaining configuration information of the newly added sharding node; updating the configuration file of the database cluster according to the configuration information so that the updated configuration file includes the configuration information; creating local tables and distributed tables on the newly added sharding node.
[0007] Optionally, in the method as described above, the database cluster includes multiple storage nodes, and after creating local tables and distributed tables on the newly added sharding nodes, it also includes: performing data balancing on the database cluster so that the data in the database cluster is evenly distributed among the storage nodes and the newly added sharding nodes.
[0008] Optionally, the method as described above performs data balancing on the database cluster, including: obtaining the data ID of the data in each storage node; obtaining the hash value of each data ID; storing the target data to the newly added shard node; the target data is data whose hash value is within a set numerical range.
[0009] Optionally, in the method described above, the database cluster is deployed as a container in a Serverless platform.
[0010] Optionally, the method as described above further comprises: cleaning the data stored in the database cluster at intervals of a preset cleaning time period.
[0011] In the second aspect, an embodiment of the present application provides a device for expanding the capacity of a database cluster, including: a monitoring module, used to obtain the storage space utilization rate and the data storage growth rate of the database cluster; the data storage growth rate is the data growth rate within a set time; the set time is the time period corresponding to the maximum data growth amount within a preset time period; if the data storage growth rate is greater than the set growth rate, and the storage space utilization rate is greater than or equal to the set space utilization rate, the storage water level control module is used to expand the capacity of the database cluster.
[0012] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;
[0013] The memory stores computer-executable instructions;
[0014] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.
[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.
[0017] The method, device, electronic device, storage medium and program product for expanding the capacity of a database cluster provided by the embodiments of the present application obtain the storage space utilization rate and data storage growth rate of the database cluster, and then automatically expand the capacity of the database cluster when the data storage growth rate is greater than the set growth rate and the storage space utilization rate is greater than or equal to the set space utilization rate. In this way, the storage situation of the database cluster can be determined based on the storage space utilization rate and the data storage growth rate. Further, based on the storage situation, it is decided whether to expand the capacity of the database cluster, so as to achieve the effect of dynamically adjusting storage resources according to actual data growth. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0019] Figure 1 A flowchart of a method for expanding the capacity of a database cluster provided in this application;
[0020] Figure 2 A flowchart of another method for expanding the capacity of a database cluster provided in this application;
[0021] Figure 3 Schematic diagram of the ClickHouse cluster architecture provided for this application;
[0022] Figure 4 A schematic diagram of the structure of the device for expanding the capacity of a database cluster provided in this application;
[0023] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.
[0024] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0026] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0027] The existing methods for expanding the capacity of database clusters are mainly based on container orchestration systems, such as Kubernetes. However, to expand the database cluster in Kubernetes, complex configuration files need to be written manually. In addition, the prerequisite for expansion is also manually triggered or dependent on a timing script, making it impossible to dynamically adjust the storage resources of the ClickHouse cluster according to actual data growth.
[0028] Based on this, the embodiment of the present application provides a method for expanding the capacity of a database cluster, by obtaining the storage status of the database cluster and then determining the timing of expanding the database based on the storage status. In this way, the effect of dynamically adjusting storage resources according to actual data growth can be achieved.
[0029] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0030] Figure 1 A flowchart of a method for expanding the capacity of a database cluster provided in this application is shown in FIG. Figure 1 As shown, the method includes:
[0031] Step S101, obtaining the storage space usage rate and data storage growth rate of the database cluster.
[0032] A database cluster is a working mode that organizes multiple database instances together. Database clusters can improve system performance, availability, and scalability. Each database instance in a database cluster is called a storage node. Data is usually stored in shards on different storage nodes.
[0033] Step S102, if the data storage growth rate is greater than the set growth rate, determine whether the storage space usage rate is greater than or equal to the set space usage rate. If not, execute step S103, if yes, execute step S106.
[0034] The data storage growth rate refers to the percentage or absolute value of the growth of the amount of data in the database cluster within a set time. The storage space utilization rate of the database cluster refers to the ratio of the space occupied by the data stored on each storage node in the database cluster to the total available storage space. Therefore, the storage space utilization rate can represent the usage status of the database cluster. The set time is the time period corresponding to the maximum data growth within the preset time period.
[0035] Within a preset time period, the amount of data storage is collected at each data collection time interval; the data collection time interval is, for example, 1 hour, 1 day, or 1 week. Calculate the data growth within each data collection time interval. Determine the time period corresponding to the maximum data growth, and determine the time period corresponding to the maximum data growth as the set time. If the database cluster is used to store work data. The preset time period is 1 day, then within the preset time period, the set time is, for example, 14:00 to 15:00. If the database cluster is used to store entertainment data, the preset time period is one month, then within the preset time period, the set time is, for example, every weekend.
[0036] The set space usage rate and the set growth rate may be set in advance or may be temporarily specified by the user.
[0037] Specifically, a space usage rate setting instruction and / or a data storage growth rate setting instruction sent by a user may be received, and then a set space usage rate is obtained from the space usage rate setting instruction and / or a set growth rate is obtained from the data storage growth rate setting instruction.
[0038] Step S103, sending a prompt message to a preset user terminal, the prompt message is used to prompt the user that the data storage growth rate of the database cluster is abnormal. Then, step S104 is executed.
[0039] Step S104, predicting the time period when the storage space usage rate reaches the set space usage rate based on the data storage growth rate. Then, executing step S105.
[0040] The set space usage can be obtained by multiplying the set space usage by the total available storage space. The storage space usage is obtained by multiplying the storage space usage by the total available storage space. Then, the difference between the set space usage and the storage space usage is obtained, and the difference is divided by the data storage growth rate to obtain the time period when the storage space usage reaches the set space usage.
[0041] Step S105, expand the capacity of the database cluster after a preset time period, and then end.
[0042] In this way, the storage situation of the database cluster can be determined based on the storage space utilization rate and the data storage growth rate. At the same time, the time period when the storage space utilization rate reaches the set space utilization rate is predicted based on the data storage growth rate. In this way, the timing of expanding the database cluster can be determined, and the database cluster can be expanded in time to reduce the situation of data storage failure caused by delayed expansion.
[0043] Step S106, expand the capacity of the database cluster, and then end.
[0044] The data stored in the database cluster can be cleaned up at intervals of a preset cleaning time period. Furthermore, data whose storage time exceeds the preset cleaning time period is deleted from the database cluster. In this way, the purpose of expanding the storage space in the database cluster can be achieved.
[0045] The preset cleaning time period may be set in advance. Alternatively, an instruction for setting the cleaning time sent by a user may be received, and the preset cleaning time period is obtained from the instruction for setting the cleaning time. The preset cleaning time period may be any value greater than or equal to 0. If the preset cleaning time period is equal to 0, it indicates that the data stored in the database cluster is not cleaned.
[0046] The method for expanding the capacity of a database cluster provided by the embodiment of the present disclosure obtains the storage space usage rate and the data storage growth rate of the database cluster, and then automatically expands the capacity of the database cluster when the data storage growth rate is greater than the set growth rate and the storage space usage rate is greater than or equal to the set space usage rate. In this way, the storage situation of the database cluster can be determined based on the storage space usage rate and the data storage growth rate. Further, whether to expand the capacity of the database cluster is determined based on the storage situation, so as to achieve the effect of dynamically adjusting storage resources according to actual data growth.
[0047] Furthermore, the database cluster is deployed as a container in the Serverless platform. The Serverless platform is a cloud computing service that allows developers to focus on writing and deploying code without having to worry about the management, configuration, and maintenance of the underlying servers. The management and operation of the server are entirely the responsibility of the cloud service provider. Since the Serverless platform is usually event-driven, the code is executed in response to specific events. This model enables applications to efficiently handle asynchronous tasks and event streams. Therefore, if the specific event is that the data storage growth rate is greater than the set growth rate, and the storage space utilization rate is greater than or equal to the set space utilization rate, the code can respond to the specific event and automatically expand the database cluster. There is no need to manually expand the scale of the database cluster, which improves the response speed and expansion capability of the database cluster.
[0048] Figure 2The flowchart of the method for expanding the capacity of a database provided in this embodiment is as follows. Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, the method for expanding the capacity of the database cluster is described in detail. The method for expanding the capacity of the database shown in this embodiment may include the following steps:
[0049] Step S201, determine whether the storage space usage is less than the maximum set usage; if so, execute step S202. Otherwise, execute step S204.
[0050] The maximum set usage rate of a database cluster refers to the ratio of the maximum available storage space in the database cluster to the total available storage space. To avoid service interruption or performance degradation caused by excessive resource consumption, the maximum available storage space is usually less than the total available storage space. The total available storage space in a database cluster can be changed. By configuring the total available storage space according to actual data storage needs, the effect of reducing resource waste can be achieved.
[0051] Step S202, expand the storage capacity of each storage node in the database cluster, and then execute step S203.
[0052] Specifically, the database cluster also includes replica nodes corresponding to each storage node. The replica node is used to back up the data stored in the corresponding storage node. The storage space of the replica node is the same as the storage space of its corresponding storage node. In the case of storage expansion of each storage node, it is also necessary to expand the replica nodes corresponding to each storage node. In the event of a storage node failure, its corresponding replica node can be used to continue to provide services, such as reading data, storing data and other services. Thereby improving the fault tolerance of the database cluster. In addition, the operation of reading data can also be dispersed to the replica nodes corresponding to the storage node, thereby reducing the load of the storage node and improving the overall performance and response speed of the database cluster. Here, there is no limit on the number of replica nodes corresponding to each storage node. It can be 1 or more. The more replica nodes there are, the more the load on the storage node can be reduced.
[0053] The newly added storage capacity can be determined by the storage space utilization rate and the data storage growth rate, and then the configuration information of each storage node can be modified according to the newly added storage capacity, thereby increasing the storage capacity of each storage node and realizing the storage expansion of each storage node.
[0054] Further, the newly added storage capacity is determined by the storage space usage rate and the data storage growth rate, including: multiplying the storage space usage rate by the total available storage space to obtain the current used space of the database cluster. Multiplying the data storage growth rate by the current used space to obtain the newly added storage capacity.
[0055] Step S203, determine whether the storage expansion is successful, if not, execute step S204. If successful, end the operation.
[0056] Among them, storage expansion may fail due to limitations of the database cluster itself or incorrect configuration of the database cluster. In this case, personnel are required to investigate the cause. However, manual investigation takes time, and the database cluster cannot be expanded in time. Based on this, the database cluster is directly sliced and expanded to ensure the timeliness of expansion.
[0057] Step S204, slice and expand the database cluster, and then execute step S205.
[0058] Specifically, the configuration information of the newly added shard node is obtained, and the configuration file of the database cluster is updated according to the configuration information, so that the updated configuration file includes the configuration information.
[0059] The configuration information of the newly added shard node includes shard information and replica information. The shard information is the basic information of the newly added shard node, and the replica information is the basic information of the replica node corresponding to the newly added shard node. Basic information includes node ID, node address for network communication, node storage device information, node storage space size, etc.
[0060] After updating the configuration file of the database cluster according to the configuration information, the method further includes: updating the configuration file of each storage node in the database cluster according to the updated configuration file, so that other nodes in the database cluster can obtain the basic information of the newly added sharding node, which facilitates the communication and interaction between the newly added sharding node and other storage nodes in the database cluster.
[0061] Furthermore, after updating the configuration file of the database cluster according to the configuration information, it also includes: creating a local table and a distributed table on the newly added sharding node. Among them, the local table on the newly added sharding node is used to store data allocated to the newly added sharding node. The distributed table on the newly added sharding node is used for the timed sharding strategy. The sharding strategy is used to define the conditions that the data stored in the newly added sharding node must meet.
[0062] Step S205: balancing data in the database cluster.
[0063] You can obtain the sharding strategy from the distributed table of the newly added sharding node and balance the data in the database cluster according to the sharding strategy.
[0064] Specifically, the sharding strategy is, for example, storing the target data in the newly added sharding node. The target data is, for example, data whose data ID hash value is within a set value range.
[0065] Alternatively, the average amount of stored data may be determined, and then the amount of data to be removed from each storage node may be determined based on the average amount of stored data; finally, the data to be removed may be determined from each storage node based on the amount of data to be removed, and the removed data may be used as the target data. Furthermore, the average amount of stored data may be obtained by dividing the total amount of data stored in the data cluster by the total number of nodes. The amount of data to be removed from each storage node may be obtained by subtracting the average amount of stored data from the amount of stored data on each storage node.
[0066] According to the order of data storage and the amount of data to be removed, the data to be removed is determined from each storage node as the target data. Further, according to the order of data storage from front to back, the data to be removed corresponding to the amount of data to be removed is selected from each storage node as the target data.
[0067] The method for expanding the capacity of a database cluster provided by the embodiment of the present disclosure is adopted. By determining whether the storage space usage rate is less than the maximum set usage rate, and then directly expanding the storage capacity of each storage node in the database cluster when the storage space usage rate is less than the maximum set usage rate. Since the total available storage space is configured according to the actual data storage demand, when the actual data storage demand increases, the total available storage space of the node can be reconfigured to increase the total available storage space of the node, that is, the storage capacity of each storage node in the database cluster is expanded. In this way, when the maximum set usage rate remains unchanged, the maximum available degree in the database cluster will increase due to the increase in the total available storage space. Thereby achieving the effect of expanding the capacity of the database cluster. In addition, by directly slicing and expanding the capacity of the database cluster when the storage space usage rate is greater than or equal to the maximum set usage rate, since the slice expansion can be increased infinitely in theory, there will be no expansion failure. Therefore, directly slicing and expanding the capacity of the database cluster can avoid data overflow caused by expansion failure.
[0068] For example, the database cluster is a ClickHouse cluster. ClickHouse is a column-based database management system for online analytical processing, which is widely popular for its superior performance in processing large-scale data sets. The ClickHouse cluster is deployed in the Serverless platform. The lightweight deployment of the ClickHouse cluster can use operator, helm or scripts to deploy the ClickHouse cluster in a container. Figure 3 As shown, Figure 3 A schematic diagram of the architecture of the ClickHouse cluster provided in the embodiment of the present disclosure. Figure 3As can be seen, the ClickHouse cluster is deployed in the Serverless platform. The ClickHouse cluster includes N ClickHouse nodes, namely ClickHouse1, ClickHouse2, ... and ClickHouseN. The Serverless platform includes a monitoring module and a storage water level control module. The monitoring module is used to obtain the storage space usage rate and data storage growth rate of the ClickHouse cluster. The storage water level control module is used to expand the database cluster when the data storage growth rate is greater than the set growth rate and the storage space usage rate is greater than or equal to the set space usage rate.
[0069] Since the existing method of expanding the capacity of the ClickHouse cluster is mainly based on the container orchestration system, such as Kubernetes and the distributed architecture of the ClickHouse cluster to achieve expansion, but deploying and expanding the ClickHouse cluster in Kubernetes requires manually writing complex configuration files and manually expanding the scale of the ClickHouse cluster, which is cumbersome and error-prone. Manual intervention is inefficient, and timing scripts are difficult to cope with sudden changes in data volume. At the same time, the traditional way of deploying ClickHouse in Kubernetes requires pre-allocation of fixed computing resources for ClickHouse. Regardless of how the business load changes, these resources will be occupied, resulting in resource waste. The prerequisite for expansion is often manually triggered or dependent on timing scripts, so that the storage resources of the ClickHouse cluster cannot be dynamically adjusted according to the actual data growth. The method for expanding the capacity of a database cluster provided by the embodiment of the present disclosure is adopted, by deploying the ClickHouse cluster in a Serverless platform. Since the Serverless platform is usually event-driven, the code is executed in response to a specific event. Therefore, if the specific event is that the data storage growth rate is greater than the set growth rate, and the storage space utilization rate is greater than or equal to the set space utilization rate, the code can respond to the specific event and automatically expand the ClickHouse cluster. There is no need to manually expand the size of the ClickHouse cluster, which improves the response speed and expansion capability of the ClickHouse cluster. At the same time, by expanding the capacity of the ClickHouse cluster when the data storage growth rate is greater than the set growth rate and the storage space utilization rate is greater than or equal to the set space utilization rate, the storage resources of the ClickHouse cluster can be dynamically adjusted according to the real-time storage data growth, improving the stability and reliability of the ClickHouse cluster.
[0070] Figure 4 A schematic diagram of the structure of the device for expanding the capacity of a database cluster provided in this application, such as Figure 4As shown, the device 400 for expanding the capacity of a database cluster provided in this embodiment includes a monitoring module 401 and a storage water level control module 402. The monitoring module 401 is used to obtain the storage space utilization rate and the data storage growth rate of the database cluster. The data storage growth rate is the data growth rate within a set time; the set time is the time period corresponding to the maximum data growth amount within a preset time period. The storage water level control module 402 is used to expand the capacity of the database cluster when the data storage growth rate is greater than the set growth rate and the storage space utilization rate is greater than or equal to the set space utilization rate.
[0071] In one possible implementation, the database cluster includes storage nodes and replica nodes corresponding to each storage node; the database cluster is expanded, including: if the storage space utilization rate is greater than the set space utilization rate and less than the maximum set utilization rate, the storage capacity of each storage node and each replica node in the database cluster is expanded; or, if the storage space utilization rate is equal to the maximum set utilization rate, the database cluster is sliced and expanded.
[0072] In one possible implementation, slicing and expanding a database cluster includes: obtaining configuration information of a newly added sharding node; updating a configuration file of the database cluster according to the configuration information so that the updated configuration file includes the configuration information; and creating a local table and a distributed table on the newly added sharding node.
[0073] In a possible implementation, after creating the local table and the distributed table on the newly added sharding node, the method further includes: balancing the data in the database cluster so that the data in the database cluster is evenly distributed in each storage node and the newly added sharding node.
[0074] In a possible implementation, data balancing is performed on a database cluster, including: obtaining data IDs of data in each storage node; obtaining hash values of each data ID; storing target data in a newly added shard node; and the target data is data whose hash values are within a set value range.
[0075] In one possible implementation, the database cluster is deployed as a container in the Serverless platform.
[0076] In a possible implementation, the device for expanding the capacity of a database cluster further includes a cleaning module, which is used to clean up the data stored in the database cluster at intervals of a preset cleaning time period.
[0077] The device for expanding the capacity of a database cluster provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and are not described in detail in this embodiment.
[0078] Figure 5This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.
[0079] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.
[0080] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0081] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.
[0082] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.
[0083] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.
[0084] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0085] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0086] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0087] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0088] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0090] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0091] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0092] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0093] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A method for expanding the capacity of a database cluster, characterized in that: include: Get the storage space usage rate and data storage growth rate of the database cluster. The data storage growth rate is the data growth rate within a set time. The set time is the time period corresponding to the maximum data growth amount within the preset time period; If the data storage growth rate is greater than the set growth rate, and the storage space usage rate is greater than or equal to the set space usage rate, the database cluster is expanded.
2. The method according to claim 1, characterized in that The database cluster includes storage nodes and replica nodes corresponding to each of the storage nodes; Expanding the database cluster includes: If the storage space usage rate is greater than the set space usage rate and less than the maximum set usage rate, then the storage capacity of each storage node and each replica node is expanded; or, If the storage space usage rate is equal to the maximum set usage rate, the database cluster is sliced and expanded.
3. The method according to claim 2, characterized in that Slicing and expanding the database cluster includes: Get the configuration information of the newly added shard node; updating a configuration file of the database cluster according to the configuration information so that the updated configuration file includes the configuration information; Create local tables and distributed tables on the newly added shard nodes.
4. The method according to claim 3, characterized in that: The database cluster includes multiple storage nodes, and after creating a local table and a distributed table on the newly added shard node, further includes: The database cluster is subjected to data balancing so that the data in the database cluster is evenly distributed in each of the storage nodes and the newly added shard nodes.
5. The method according to claim 4, characterized in that Performing data balancing on the database cluster includes: Obtaining the data ID of the data in each of the storage nodes; Obtaining a hash value of each of the data IDs; The target data is stored in the newly added shard node; the target data is data whose hash value is within a set value range.
6. The method according to claim 1, characterized in that The database cluster is deployed as a container in the Serverless platform.
7. The method according to claim 1, characterized in that Also includes: The data stored in the database cluster is cleaned up at intervals of a preset cleaning time period.
8. A device for expanding the capacity of a database cluster, characterized in that: include: Monitoring module, used to obtain the storage space usage rate and data storage growth rate of the database cluster; The data storage growth rate is the data growth rate within a set period of time; The set time is the time period corresponding to the maximum data growth amount within the preset time period; If the data storage growth rate is greater than the set growth rate, and the storage space usage rate is greater than or equal to the set space usage rate, the storage water level control module is used to expand the database cluster.
9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.
11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.
Citation Information
Patent Citations
Data space management method and device
CN110825308A
Container cluster scheduling method and device, equipment and storage medium
CN116340005A
Storage cluster capacity expansion method and device, computer equipment and storage medium
CN117435129A
