Deployment and distribution method and system based on distributed database system instances

Abstract

The invention discloses a deployment and distribution method and system based on a distributed database system instance, which are used for optimizing and perfecting a distributed deployment process based on a database system and realizing a distributed deployment and distribution algorithm. According to characteristics and functional characteristics of a component architecture in a distributed database instance and evaluation of resource use and service load conditions of an equipment server, each component of a distributed database system is reasonably allocated and deployed; according to the method, the high availability of the instance of the distributed database system is guaranteed, meanwhile, the resource use condition of the component on each deployment server is optimized, the planning and deployment process of the distributed storage system is simplified, and the resource consumption of the server caused by the instance of the distributed database in the service is reduced.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a deployment and allocation method and system based on a distributed database system instance. Background Art

[0002] To maintain high availability and stability during distributed system deployment, two key requirements must be adhered to: cross-isolation deployment and rational resource allocation. The former requires distributed components to be distributed across limited devices as much as possible to avoid single points of failure associated with centralized deployment. In situations where system reliability is even more critical, distributed components often need to be deployed across different data centers to further disperse risk and improve disaster recovery. Furthermore, when implementing distributed redundant deployments, the resource availability of each server must be considered during deployment to balance the load across servers and improve the performance and stability of the distributed system.

[0003] When we deploy distributed database system instances on multiple device servers, we often face the following problems:

[0004] Complex component roles: The distributed architecture of the database system includes a large number of replica services and is composed of several components with different functions, such as shard storage services, metadata services, monitoring services, and computing services. The complex component structure often leads to more stringent deployment rules and allocation requirements. For example, in actual deployment, the main library of the storage service component should not be deployed on the same device server as the main library of the storage service of other shards to ensure the independence of data shards. At the same time, the storage service backup library should not be deployed on the same device server as the main library of the storage service of each shard. Otherwise, storage and backup will not be able to guarantee the high availability of the database instance in the event of a device server failure.

[0005] System resource imbalance: In the scenario of distributed database storage, an imperfect deployment plan may affect the high availability of the system. If key components are concentrated on a few servers, once a failure occurs, it will have a serious impact on the normal operation of the entire system. At the same time, uneven distribution of server resources may cause some servers to be overloaded while other server resources are idle, which not only affects the overall performance of the system but also leads to waste of system resources. Summary of the Invention

[0006] Purpose of the invention: The present invention provides a deployment and allocation method and system based on a distributed database system instance, which not only ensures the high availability of the distributed database instance but also optimizes the resource usage of components on each deployment server, simplifies the planning and deployment process of the distributed storage system, and reduces the resource consumption of the server caused by the distributed database instance in the business.

[0007] Technical solution: The present invention provides a deployment and allocation method based on a distributed database system instance, comprising the following steps:

[0008] Step 1: Iterate through each component in the distributed architecture information to be deployed;

[0009] Step 2: Evaluate the functional roles of the components iterated in step 1 and filter out unsuitable target servers according to the specific distributed architecture rules of different components.

[0010] Step 3: Get the distribution of currently allocated components, calculate the available memory in the current iteration for the devices in the list, and sort them in descending order, so that devices with more available memory are arranged earlier in the list to be allocated;

[0011] Step 4: Based on the distribution of currently allocated components, the devices in the list are sorted again in ascending order according to the number of components of the same type that have been allocated in the current iteration, so that devices with fewer components of the same type that have been allocated are arranged earlier in the list to be allocated.

[0012] Step 5: If the component type of the current iteration is a metadata service component or it is the first time that this type of component is allocated in this distributed cluster, the first device server in the list to be allocated is directly selected for allocation, the server allocation status is updated, the subsequent steps are skipped, and the next component iteration is entered; otherwise, the process proceeds to step 6;

[0013] Step 6: If the component of the current iteration has been allocated to a similar component before the current iteration, then obtain the information of the computer room to which the device server to which the similar component has been allocated belongs;

[0014] Step 7: Filter the list of devices to be allocated, remove all devices in the same computer room that have been allocated the same component, and obtain a list of device options that are in different computer rooms from the device servers that have been allocated the same component. If the list of device options is not empty, select the first device server in the order as the deployment target, update the server allocation, skip the subsequent steps, and proceed to the next component iteration. Otherwise, proceed to step 8.

[0015] In step 8, if the device option list after filtering by the computer room screening criteria in step 7 is empty, it means that the same type of components have been allocated in all computer rooms. At this time, obtain the list of devices to be allocated before filtering in step 7, filter out the devices that have been allocated with the same type of components, and obtain the device option list that has not been deployed with the same type of components. If the device option list is not empty, select the first device server in order as the deployment target, update the server allocation status, skip the subsequent steps and proceed to the next component iteration. Otherwise, proceed to step 9.

[0016] If the device option list after filtering by the device screening criteria in steps 9 and 8 is empty, it means that the same type of components have been allocated to all devices. At this time, obtain the list to be allocated before filtering in step 7 and directly select the first device server as the deployment target in order, update the server allocation status, and enter the next component iteration; after the above operations, the allocation of distributed architecture components for the current iteration is completed.

[0017] Furthermore, in step 1, each component in the distributed architecture information to be deployed is iterated in the order of "metadata service component, computing service component, monitoring service component, shard 1 storage component main library, shard 1 storage component backup library, shard 2 storage component main library, shard 2 storage component backup library..."

[0018] Furthermore, in step 2, if the current component is a metadata service component, no processing is performed; if it is a sharded storage component master library type, the servers allocated to the master libraries of other shards are obtained and filtered out from the candidate list; if it is a sharded storage component backup library type, the servers allocated to the master library of the current shard are obtained and filtered out from the candidate list; if it is other types, the metadata service component and the servers allocated to the same type of components as the current one are obtained and filtered out from the candidate list.

[0019] Furthermore, in step 2, if after preliminary screening, the elements in the list are empty, it means that all optional device servers have been assigned at least one component, and this iteration obtains all deployable servers as the candidate list.

[0020] Furthermore, in step 3, the available memory is the actual available memory - the number of allocated components * the estimated component memory usage.

[0021] Furthermore, in step 5, when each type of component is allocated for the first time, since there is no need to consider the distribution of similar components, the first item can be directly selected from the list of servers to be allocated, that is, the server with the largest memory can be used for allocation.

[0022] Correspondingly, a deployment and distribution system based on a distributed database system instance includes: a metadata service component, a monitoring service component, a computing service component and a storage service component; the metadata service component stores the node information and partition information metadata of the storage system, accepts metadata information requests from other components, and also performs data scheduling and resource load balancing for the entire cache database according to the system load; the storage service component is a node unit for data storage, responsible for providing all core data reading and writing services for the entire database; the computing service component distributes user requests to the storage server of the corresponding shard through hash calculation, and feeds back the return results of the storage server to external applications; the monitoring service component monitors the primary and backup libraries of each shard in the storage cluster and provides automatic fault handling functions.

[0023] Furthermore, computing service components should be avoided from being deployed on the same device server as other computing service components to share the load on the server caused by user requests, improve the availability of database services, and reduce the impact of server downtime on database instance request processing; at the same time, computing service components should be avoided from being deployed on the same device server as metadata service components to minimize the mutual impact between components.

[0024] Furthermore, the monitoring service component monitors the primary and backup databases of each shard in the storage cluster and provides automatic fault handling. If a primary database failure is detected, all monitoring service components within the instance will coordinate and elect a backup database as the new primary database to ensure the cluster continues to provide services. In actual deployment, the monitoring service component should not be deployed on the same device server as other components of the same type to prevent simultaneous failures of multiple monitoring service components, which could lead to a loss of primary / backup storage service switchover. Cross-deployment of monitoring service components can improve the disaster recovery capabilities of the database instance.

[0025] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: it optimizes and improves the distributed deployment process based on the database, realizes the distributed deployment allocation algorithm, and reasonably allocates and deploys the various components of the distributed database system according to the characteristics and functional characteristics of the component architecture in the distributed database instance and evaluates the resource usage and service load of the device server. While ensuring the high availability of the distributed database system instance, it also optimizes the resource usage of the components on each deployment server, simplifies the planning and deployment process of the distributed storage system, and reduces the resource consumption of the server caused by the distributed database instance in the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the system architecture of the present invention.

[0027] Figure 2 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0028] like Figure 1 As shown, a deployment and allocation system based on a distributed database system instance includes: a metadata service component, a storage service component cluster, a computing service component cluster, and a monitoring service component cluster.

[0029] The metadata service component primarily stores system metadata, including node and partition information. It accepts metadata requests from other components and also schedules data and balances resource load across the cache database based on system load. It is the most important core component of a distributed database system.

[0030] The Storage Service component cluster is a node unit for data storage, responsible for providing all core data services, including read and write services, for the entire database. In a distributed cluster, data is distributed across different storage servers using hash value range sharding to increase the cluster's storage capacity and processing power. Based on their specific roles and functions, the Storage Service component can be divided into primary and secondary databases. The primary database is the main node providing external services, accepting client read and write requests and synchronously sending data to the secondary database. The secondary database receives data backups from the primary database and can accept client read requests, thus reducing the workload on the primary database. Due to the different roles and functions of the two types of Storage Service components, in actual deployment, the Storage Service primary database should not be deployed on the same device server as the Storage Service primary databases of other shards. This ensures the independence of data shards, allowing each shard to fully utilize its own server's system resources and improve data storage efficiency and capacity. Furthermore, the Storage Service secondary database should not be deployed on the same device server as the Storage Service primary database of its respective shard. Otherwise, storage and backup services will be unavailable in the event of a device server failure, making it impossible to ensure high availability of the database instance.

[0031] The compute service component cluster does not store any user data. Instead, it distributes user requests to the corresponding shard's storage server through hash calculations and feeds the storage server's return results back to external applications. In actual deployments, compute components should not be deployed on the same device server as other compute service components to share the server load from user requests, improve database service availability, and reduce the impact of server downtime on database instance request processing. Compute components should also be deployed on the same device server as metadata service components to minimize interactions between components.

[0032] The monitoring service component is responsible for monitoring the primary and backup databases of each shard in the storage cluster and providing automatic fault handling. When a primary database failure is detected, all monitoring service components within the instance will negotiate and coordinate to elect a backup database as the new primary database to ensure the cluster can continue to provide services. During actual deployment, the monitoring service component should not be deployed on the same device server as other components of the same type to prevent simultaneous failures of multiple monitoring service components, which could cause the storage service to lose its primary and backup switching capabilities. Cross-deployment of monitoring service components can improve the disaster recovery capabilities of the database instance. Compute components should also be deployed on the same device server as metadata service components to minimize the impact of each component on the other.

[0033] like Figure 2 As shown, a deployment allocation method based on a distributed database system instance includes the following steps:

[0034] Step 1. Iterate and traverse each component in the distributed architecture information to be deployed in the order of "metadata service component, computing service component, monitoring service component, shard 1 storage component main library, shard 1 storage component backup library, shard 2 storage component main library, shard 2 storage component backup library..."

[0035] Step 2: Evaluate the functional roles of the components iterated in step 1, and filter out unsuitable target servers according to the specific distributed architecture rules of different components. If the current component is a metadata service component, no processing is performed; if it is a sharded storage component master library type, obtain the servers assigned to the master libraries of other shards and filter them out from the candidate list; if it is a sharded storage component backup library type, obtain the servers assigned to the master library of the current shard and filter them out from the candidate list; if it is other types, obtain the metadata service component and the servers assigned to the current component of the same type and filter them out from the candidate list; after the above operations, the candidate list of deployable servers is preliminarily screened so that the allocation plan generated by the algorithm complies with the distributed architecture rules, and conflicts and interference that may arise from the distribution of components are avoided when sufficient server resources are allocated.

[0036] Step 3: Get the distribution of currently allocated components, calculate the available memory in the current iteration for the devices in the list (actual available memory - number of allocated components * estimated component memory usage) and sort them in descending order.

[0037] Step 4. Based on the distribution of currently allocated components, the devices in the list are sorted again in ascending order according to the number of components of the same type that have been allocated in the current iteration. After the above operations, the candidate list of deployable servers is sorted according to available memory and server load. In the subsequent allocation process, devices with more available memory and less server load will be given priority for allocation, ensuring that component deployment fully utilizes server resources and maintains the load balance of the server at the component deployment level.

[0038] Step 5: If the component type of the current iteration is a metadata service component or it is the first time that this type of component is allocated in this distributed cluster, then directly select the first device server in the list to be allocated for allocation, update the server allocation status, skip the subsequent steps and enter the next component iteration (starting from step 2), otherwise enter step 6.

[0039] Step 6: If it is determined that a component of the current iteration has been allocated to a component of the same type before the current iteration, then the information of the computer room to which the device server to which the component of the same type has been allocated is obtained.

[0040] Step 7: Filter the list to be allocated, remove all devices in the computer room that have been allocated the same type of components, and obtain a list of device options that are in different computer rooms from the device servers that have been allocated the same type of components. If the device option list is not empty, select the first device server in order as the deployment target, update the server allocation status, skip subsequent steps and enter the next component iteration (starting from step 2), otherwise enter step 8.

[0041] In step 8, if the device option list after filtering by the computer room screening criteria in step 7 is empty, it means that the same type of components have been allocated in all computer rooms. At this time, obtain the list to be allocated before filtering in step 7, filter out the devices that have been allocated with the same type of components, and obtain the device option list where the same type of components have not been deployed. If the device option list is not empty, select the first device server in order as the deployment target, update the server allocation status, skip the subsequent steps and enter the next component iteration (starting from step 2), otherwise enter step 9.

[0042] If the device option list after filtering by the device selection criteria in steps 9 and 8 is empty, it means that the same component has already been allocated to all devices. At this time, the list of pending allocations before filtering in step 7 is obtained and the first device server is directly selected in order as the deployment target. The server allocation is updated and the next component iteration begins (starting from step 2). After the above operations, the allocation of the distributed architecture components of the current iteration is completed, ensuring that the iterated components comply with the database distributed architecture rules under the current circumstances and that the components are cross-distributed across the device servers, maintaining the high availability of the components. When the iteration of the components in the database distributed architecture is completed, the algorithm program outputs the corresponding database distributed instance deployment allocation plan.

[0043] In step 2, the specific order and rules of screening are as follows:

[0044] Metadata service component: Deployed independently, without filtering the device server.

[0045] Compute service components: They need to be cross-deployed with metadata service components and other compute service components. Therefore, device server options that have been assigned metadata service components and compute service components are excluded.

[0046] Monitoring service components: They need to be cross-deployed with metadata service components and other monitoring service components. Therefore, the device server options that have been assigned metadata service components and monitoring service components are filtered out.

[0047] Shard storage component master library: It needs to be cross-deployed with other shard storage component master libraries, so the device server options that have been assigned other shard storage component master libraries are filtered out.

[0048] Shard storage component backup database: It needs to be deployed cross-platform with the primary database in this shard. It cannot be deployed on the same device as the primary database of the same shard. Therefore, the device server option that has been assigned the primary database of this shard storage component is filtered out.

[0049] In step 2, if the list is empty after the initial screening, it means that all available device servers have been assigned at least one component, and this iteration obtains all deployable servers as the candidate list. This step aims to evenly distribute components to idle devices, minimizing the deployment of similar components on the same device. Furthermore, since sharded storage component backups cannot be deployed on the same device as the primary database for the same shard, the backup database does not exclude other backup nodes during the screening process. Therefore, the list obtained after the initial screening will never be empty, eliminating the possibility of re-acquiring the primary database device for the same shard.

[0050] In step 3, the available memory of the device server is calculated by subtracting the reserved memory of the allocated components on the device from the current available memory, to prevent a large number of components from being allocated on a server with more available memory.

[0051] In step 5, when each type of component is allocated for the first time, since there is no need to consider the distribution of similar components, the first item can be directly selected from the list of servers to be allocated, that is, the server with the largest memory can be used for allocation.

[0052] The present invention provides a deployment allocation algorithm based on a distributed database system instance, which not only meets the deployment rules of the database distributed architecture and the high availability requirements of cross-distribution, but also can solve the problem of imbalance in resource consumption and component ratio by deploying and allocating to target servers with different resource margins. Through this invention, the integrity and high availability of the deployed distributed database system cluster can be guaranteed, and the resource allocation structure of the deployment server can be optimized.

Claims

1. A deployment and allocation method based on a distributed database system instance, characterized in that: The steps include: Step 1: Iterate through each component in the distributed architecture information to be deployed; Step 2: Evaluate the functional roles of the components iterated in step 1 and filter out unsuitable target servers according to the specific distributed architecture rules of different components. Step 3: Get the distribution of currently allocated components, calculate the available memory in the current iteration for the devices in the list, and sort them in descending order, so that devices with more available memory are arranged earlier in the list to be allocated; Step 4: Based on the distribution of currently allocated components, the devices in the list are sorted again in ascending order according to the number of components of the same type that have been allocated in the current iteration, so that devices with fewer components of the same type that have been allocated are arranged earlier in the list to be allocated. Step 5: If the component type of the current iteration is a metadata service component or it is the first time that this type of component is allocated in this distributed cluster, the first device server in the list to be allocated is directly selected for allocation, the server allocation status is updated, the subsequent steps are skipped, and the next component iteration is entered; otherwise, the process proceeds to step 6; Step 6: If the component of the current iteration has been allocated to a similar component before the current iteration, then obtain the information of the computer room to which the device server to which the similar component has been allocated belongs; Step 7: Filter the list of devices to be allocated, remove all devices in the same computer room that have been allocated the same component, and obtain a list of device options that are in different computer rooms from the device servers that have been allocated the same component. If the list of device options is not empty, select the first device server in the order as the deployment target, update the server allocation, skip the subsequent steps, and proceed to the next component iteration. Otherwise, proceed to step 8. In step 8, if the device option list after filtering by the computer room screening criteria in step 7 is empty, it means that the same type of components have been allocated in all computer rooms. At this time, obtain the list of devices to be allocated before filtering in step 7, filter out the devices that have been allocated with the same type of components, and obtain the device option list that has not been deployed with the same type of components. If the device option list is not empty, select the first device server in order as the deployment target, update the server allocation status, skip the subsequent steps and proceed to the next component iteration. Otherwise, proceed to step 9. If the device option list after filtering by the device screening criteria in steps 9 and 8 is empty, it means that the same type of components have been allocated to all devices. At this time, obtain the list to be allocated before filtering in step 7 and directly select the first device server as the deployment target in order, update the server allocation status, and enter the next component iteration; after the above operations, the allocation of distributed architecture components for the current iteration is completed.

2. The deployment and allocation method based on a distributed database system instance according to claim 1, characterized in that: In step 1, the components in the distributed architecture information to be deployed are iterated in the order of "metadata service component, computing service component, monitoring service component, shard 1 storage component master library, shard 1 storage component backup library, shard 2 storage component master library, shard 2 storage component backup library..." 3. The deployment and allocation method based on a distributed database system instance according to claim 1, characterized in that: In step 2, if the current component is a metadata service component, no processing is performed; if it is a shard storage component master library type, the server assigned to the master library of other shards is obtained and filtered out from the candidate list; If it is a shard storage component backup database type, obtain the server assigned to the primary database of the current shard and filter it out from the candidate list; if it is other types, obtain the metadata service component and the server assigned to the current component of the same type and filter it out from the candidate list.

4. The deployment and allocation method based on a distributed database system instance according to claim 1, characterized in that: In step 2, if after preliminary screening, if the elements in the list are empty, it means that all optional device servers have been assigned at least one component of the same type, and this iteration obtains all deployable servers as the candidate list.

5. The deployment and allocation method based on a distributed database system instance according to claim 1, characterized in that: In step 3, the available memory is the actual available memory - the number of allocated components * the estimated component memory usage.

6. The deployment and allocation method based on a distributed database system instance according to claim 1, characterized in that: In step 5, when each type of component is allocated for the first time, since there is no need to consider the distribution of similar components, the first item can be directly selected from the list of servers to be allocated, that is, the server with the largest memory can be used for allocation.

7. A system based on the deployment and allocation method based on a distributed database system instance as claimed in claim 1, characterized in that: include: Metadata service component, monitoring service component, computing service component, and storage service component; The metadata service component stores the system's node and partition information metadata, accepts metadata information requests from other components, and performs data scheduling and resource load balancing for the entire cache database based on the system load. The storage service component is the node unit for data storage and is responsible for providing all core data read and write services for the entire database. The computing service component distributes user requests to the storage server of the corresponding shard through hash calculation, and feeds back the return results of the storage server to the external application; the monitoring service component monitors the primary and backup libraries of each shard in the storage cluster and provides automatic fault handling functions.

8. The deployment and distribution system based on a distributed database system instance according to claim 7, characterized in that: Compute service components should not be deployed on the same device server as other compute service components. Compute service components should also not be deployed on the same device server as metadata service components.

9. The deployment and distribution system based on a distributed database system instance according to claim 7, characterized in that: The monitoring service component is responsible for monitoring the primary and backup databases of each shard in the storage cluster and providing automatic fault handling capabilities: when a failure in the primary database is detected, all monitoring service components in the instance will negotiate and cooperate to elect a backup database as the new primary database. The monitoring service component should not be deployed on the same device server as other components of the same type.

Images